# embedding_metadata

key: chroma:document
string_value:
CODE:

def eval_node(node):
    if isinstance(node, ast.Num):  # <number>
        return node.n
    elif isinstance(node, ast.BinOp):  # <left> <operator> <right>
        return MATH_OPERATORS[type(node.op)](eval_node(node.left), eval_node(node.right))
    elif isinstance(node, ast.UnaryOp):  # <operator> <operand> e.g., -1
        return MATH_OPERATORS[type(node.op)](eval_node(node.operand))
    else:


SUMMARY:
**Eval Node Function Summary**

### Purpose and Functionality

The `eval_node` function is a recursive traverser of abstract syntax tree (AST) nodes, evaluating arithmetic expressions by applying mathematical operators to their operands.

### Key Implementation Details

* Evaluates integer literal nodes (`ast.Num`) as-is.
* Unifies binary operation nodes (`ast.BinOp`): recursively evaluates left and right operands with operator-dependent functions from `MATH_OPERATORS`.
* Unifies unary operation nodes (`ast.UnaryOp`): recursively evaluates the operand with an operator-dependent function from `MATH_OPERATORS`.
* Raises a `TypeError` for unsupported node types.

### Dependencies

None, as this implementation does not rely on external dependencies. However, it utilizes:

* `ast` (Abstract Syntax Tree) module for parsing expressions.
* `MATH Operators` dictionary (not shown in provided code), which maps operator types to evaluation functions.
* No external libraries besides the Python standard library.


key: chroma:document
string_value:
CODE:

def replace_default_in_arg_description(description: str, default: Any) -> str:
    """
    Catches the default value in the description of an argument inside a docstring and replaces it by the value passed.

    Args:
        description (`str`): The description of an argument in a docstring to process.
        default (`Any`): The default value that whould be in the docstring of that argument.

    Returns:
       `str`: The description updated with the new default value.
    """
    # Lots of docstrings have `optional` or **opational** instead of *optional* so we do this fix here.
    description = description.replace("`optional`", OPTIONAL_KEYWORD)
    description = description.replace("**optional**", OPTIONAL_KEYWORD)
    if default is inspect._empty:
        # No default, make sure the description doesn't have any either
        idx = description.find(OPTIONAL_KEYWORD)
        if idx != -1:
            description = description[:idx].rstrip()
            if description.endswith(","):
                description = description[:-1].rstrip()
    elif default is None:
        # Default None are not written, we just set `*optional*`. If there is default that is not None specified in the
        # description, we do not erase it (as sometimes we set the default to `None` because the default is a mutable
        # object).
        idx = description.find(OPTIONAL_KEYWORD)
        if idx == -1:
            description = f"{description}, {OPTIONAL_KEYWORD}"
        elif re.search(r"defaults to `?None`?", description) is not None:
            len_optional = len(OPTIONAL_KEYWORD)
            description = description[: idx + len_optional]
    else:
        str_default = None
        # For numbers we may have a default that is given by a math operation (1/255 is really popular). We don't
        # want to replace those by their actual values.
        if isinstance(default, (int, float)) and re.search("defaults to `?(.*?)(?:`|$)", description) is not None:
            # Grab the default and evaluate it.
            current_default = re.search("defaults to `?(.*?)(?:`|$)", description).groups()[0]
            if default == eval_math_expression(current_default):
                try:
                    # If it can be directly converted to the type of the default, it's a simple value
                    str_default = str(type(default)(current_default))
                except Exception:
                    # Otherwise there is a math operator so we add a code block.
                    str_default = f"`{current_default}`"
            elif isinstance(default, enum.Enum) and default.name == current_default.split(".")[-1]:
                # When the default is an Enum (this is often the case for PIL.Image.Resampling), and the docstring
                # matches the enum name, keep the existing docstring rather than clobbering it with the enum value.
                str_default = f"`{current_default}`"

        if str_default is None:
            str_default = stringify_default(default)
        # Make sure default match
        if OPTIONAL_KEYWORD not in description:
            description = f"{description}, {OPTIONAL_KEYWORD}, defaults to {str_default}"
        elif _re_parse_description.search(description) is None:
            idx = description.find(OPTIONAL_KEYWORD)
            len_optional = len(OPTIONAL_KEYWORD)
            description = f"{description[:idx + len_optional]}, defaults to {str_default}"
        else:
            description = _re_parse_description.sub(rf"*optional*, defaults to {str_default}", description)



SUMMARY:
**Summary:**

Function `replace_default_in_arg_description` updates the argument description in a docstring to include the provided default value. It performs various fixes and transformations based on the type and content of the default value.

**Key Features:**

* Replaces occurrences of "optional" or "**optional**" with `OPTIONAL_KEYWORD`.
* Handles cases where the default value is `None`, `empty string` (e.g., empty line), or a complex math expression.
* For numbers, evaluates the math expression if it matches the default value and converts it to the correct type.

**Important Variables and Functions:**

* `OPTIONAL_KEYWORD`: "optional" or "**optional**"
* `description`: input docstring
* `default`: default value to be included in the updated description

**Notable Dependencies:**

* `inspect` for checking `default` value
* `re` for regular expression matching and replacing
* `eval_math_expression` function not provided in the code snippet, but used for evaluating math expressions

**Return Value:**

The updated docstring with the default value included.


key: chroma:document
string_value:
CODE:

def get_default_description(arg: inspect.Parameter) -> str:
    """
    Builds a default description for a parameter that was not documented.

    Args:
        arg (`inspect.Parameter`): The argument in the signature to generate a description for.

    Returns:
        `str`: The description.
    """
    if arg.annotation is inspect._empty:
        arg_type = "<fill_type>"
    elif hasattr(arg.annotation, "__name__"):
        arg_type = arg.annotation.__name__
    else:
        arg_type = str(arg.annotation)

    if arg.default is inspect._empty:
        return f"`{arg_type}`"
    elif arg.default is None:
        return f"`{arg_type}`, {OPTIONAL_KEYWORD}"
    else:
        str_default = stringify_default(arg.default)


SUMMARY:
**get_default_description function summary**

```markdown
Provides default description for an uncovered argument in Python function signatures.

### Key implementation details:

* Uses `inspect.Parameter` and related types (`inspect._empty`) to determine parameter type and defaults.
* Employs string formatting to construct the final default description.

### Notable dependencies and interfaces:

* `inspect`: Imported module providing type inspection utilities.
* `optional`: Likely an external library (e.g., `pydantic`) for optional parameter handling, referenced by `OPTIONAL_KEYWORD`.
```

### Primary purpose and functionality
Builds a default description string for an uncovered argument in a Python function signature, considering its type and potential default value.


key: chroma:document
string_value:
CODE:

def find_source_file(obj: Any) -> Path:
    """
    Finds the source file of an object.

    Args:
        obj (`Any`): The object whose source file we are looking for.

    Returns:
        `Path`: The source file.
    """
    module = obj.__module__
    obj_file = PATH_TO_TRANSFORMERS
    for part in module.split(".")[1:]:
        obj_file = obj_file / part


SUMMARY:
**Summary:**

Finds the source file of an object given its `__module__` attribute.

*   **Purpose:** Returns the path to the source file associated with the input object's module.
*   **Implementation Details:**
    *   Uses the `__module__` attribute of the input object to determine the module name.
    *   Joins the module name with a prefix (`PATH_TO_TRANSFORMERS`) to form the initial directory path.
    *   Iterates through the parts of the module name (skipping the first part, which is the package name), appending each part as a directory.
        *   `module.split(".")[1:]`: Extracts the namespace from the module name after removing the package prefix ('__main__.').
    *   Returns the final path with a `.py` suffix using `Path.with_suffix()`.
*   **Dependencies:** None explicitly mentioned; however, it relies on the `PATH_TO_TRANSFORMERS` constant and possibly `Path`.

**Key Variables/Functions:**

*   `obj`: Input object with its `__module__` attribute.
*   `module`: The namespace of the input object's module name (e.g., 'transformers.modeling.tokenization').
*   `PARTS OF THE MODULE NAME`: An array consisting of module parts excluding the package index prefix (`__main__.`).


key: chroma:document
string_value:
CODE:

def match_docstring_with_signature(obj: Any) -> Optional[Tuple[str, str]]:
    """
    Matches the docstring of an object with its signature.

    Args:
        obj (`Any`): The object to process.

    Returns:
        `Optional[Tuple[str, str]]`: Returns `None` if there is no docstring or no parameters documented in the
        docstring, otherwise returns a tuple of two strings: the current documentation of the arguments in the
        docstring and the one matched with the signature.
    """
    if len(getattr(obj, "__doc__", "")) == 0:
        # Nothing to do, there is no docstring.
        return

    # Read the docstring in the source code to see if there is a special command to ignore this object.
    try:
        source, _ = inspect.getsourcelines(obj)
    except OSError:
        source = []

    idx = 0
    while idx < len(source) and '"""' not in source[idx]:
        idx += 1

    ignore_order = False
    if idx < len(source):
        line_before_docstring = source[idx - 1]
        if re.search(r"^\s*#\s*no-format\s*$", line_before_docstring):
            # This object is ignored
            return
        elif re.search(r"^\s*#\s*ignore-order\s*$", line_before_docstring):
            ignore_order = True

    # Read the signature
    signature = inspect.signature(obj).parameters

    obj_doc_lines = obj.__doc__.split("\n")
    # Get to the line where we start documenting arguments
    idx = 0
    while idx < len(obj_doc_lines) and _re_args.search(obj_doc_lines[idx]) is None:
        idx += 1

    if idx == len(obj_doc_lines):
        # Nothing to do, no parameters are documented.
        return

    indent = find_indent(obj_doc_lines[idx])
    arguments = {}
    current_arg = None
    idx += 1
    start_idx = idx
    # Keep going until the arg section is finished (nonempty line at the same indent level) or the end of the docstring.
    while idx < len(obj_doc_lines) and (
        len(obj_doc_lines[idx].strip()) == 0 or find_indent(obj_doc_lines[idx]) > indent
    ):
        if find_indent(obj_doc_lines[idx]) == indent + 4:
            # New argument -> let's generate the proper doc for it
            re_search_arg = _re_parse_arg.search(obj_doc_lines[idx])
            if re_search_arg is not None:
                _, name, description = re_search_arg.groups()
                current_arg = name
                if name in signature:
                    default = signature[name].default
                    if signature[name].kind is inspect._ParameterKind.VAR_KEYWORD:
                        default = None
                    new_description = replace_default_in_arg_description(description, default)
                else:
                    new_description = description
                init_doc = _re_parse_arg.sub(rf"\1\2 ({new_description}):", obj_doc_lines[idx])
                arguments[current_arg] = [init_doc]
        elif current_arg is not None:
            arguments[current_arg].append(obj_doc_lines[idx])

        idx += 1

    # We went too far by one (perhaps more if there are a lot of new lines)
    idx -= 1
    if current_arg:
        while len(obj_doc_lines[idx].strip()) == 0:
            arguments[current_arg] = arguments[current_arg][:-1]
            idx -= 1
    # And we went too far by one again.
    idx += 1

    old_doc_arg = "\n".join(obj_doc_lines[start_idx:idx])

    old_arguments = list(arguments.keys())
    arguments = {name: "\n".join(doc) for name, doc in arguments.items()}
    # Add missing arguments with a template
    for name in set(signature.keys()) - set(arguments.keys()):
        arg = signature[name]
        # We ignore private arguments or *args/**kwargs (unless they are documented by the user)
        if name.startswith("_") or arg.kind in [
            inspect._ParameterKind.VAR_KEYWORD,
            inspect._ParameterKind.VAR_POSITIONAL,
        ]:
            arguments[name] = ""
        else:
            arg_desc = get_default_description(arg)
            arguments[name] = " " * (indent + 4) + f"{name} ({arg_desc}): <fill_docstring>"

    # Arguments are sorted by the order in the signature unless a special comment is put.
    if ignore_order:
        new_param_docs = [arguments[name] for name in old_arguments if name in signature]
        missing = set(signature.keys()) - set(old_arguments)
        new_param_docs.extend([arguments[name] for name in missing if len(arguments[name]) > 0])
    else:
        new_param_docs = [arguments[name] for name in signature.keys() if len(arguments[name]) > 0]
    new_doc_arg = "\n".join(new_param_docs)



SUMMARY:
**Function Summary:**
### match_docstring_with_signature(obj)
 
  **Purpose:** Match the docstring of an object with its signature.

  **Primary Functionality:** The function `match_docstring_with_signature` compares two documentation strings for an object's methods and arguments. It reads the source code (if available) to check for ignore commands, then parses the signatures using `inspect.signature`. It extracts argument names and descriptions from the docstring, generating new documentations based on the found signature parameters.

  **Key Implementation Details:**
- Uses regular expressions (`re`) for pattern matching.
- Leverages `find_indent` (function not shown in this snippet) to parse indentation levels.
- Utilizes `_pre_parse_arg` and `_parse_arg` regular expression patterns (not directly visible here).

  **Dependencies/Requirements:**
- Import of `inspect`, `re`, and access to `type Any`, `Optional`, `Tuple`, and `inspect._ParameterKind`.
 - The function utilizes type hints for clarity (`Any`, `Inspect.Signature`).
  
  **API Interface:** 
    It accepts an object, returns a tuple containing two strings:
     - The first string contains the original documentation of the arguments described in the docstring.
     - The second string contains the matched and generated documentation for the same argument parameters based on their signature.

**Additional Notes:**
The function seems heavily involved with parsing and manipulating various parts of Python source code, likely to maintain consistency across different codebases by following certain established coding standards.


key: chroma:document
string_value:
CODE:

def fix_docstring(obj: Any, old_doc_args: str, new_doc_args: str):
    """
    Fixes the docstring of an object by replacing its arguments documentaiton by the one matched with the signature.

    Args:
        obj (`Any`):
            The object whose dostring we are fixing.
        old_doc_args (`str`):
            The current documentation of the parameters of `obj` in the docstring (as returned by
            `match_docstring_with_signature`).
        new_doc_args (`str`):
            The documentation of the parameters of `obj` matched with its signature (as returned by
            `match_docstring_with_signature`).
    """
    # Read the docstring in the source code and make sure we have the right part of the docstring
    source, line_number = inspect.getsourcelines(obj)

    # Get to the line where we start documenting arguments
    idx = 0
    while idx < len(source) and _re_args.search(source[idx]) is None:
        idx += 1

    if idx == len(source):
        # Args are not defined in the docstring of this object
        return

    # Get to the line where we stop documenting arguments
    indent = find_indent(source[idx])
    idx += 1
    start_idx = idx
    while idx < len(source) and (len(source[idx].strip()) == 0 or find_indent(source[idx]) > indent):
        idx += 1

    idx -= 1
    while len(source[idx].strip()) == 0:
        idx -= 1
    idx += 1

    if "".join(source[start_idx:idx])[:-1] != old_doc_args:
        # Args are not fully defined in the docstring of this object
        return

    obj_file = find_source_file(obj)
    with open(obj_file, "r", encoding="utf-8") as f:
        content = f.read()

    # Replace content
    lines = content.split("\n")
    lines = lines[: line_number + start_idx - 1] + [new_doc_args] + lines[line_number + idx - 1 :]

    print(f"Fixing the docstring of {obj.__name__} in {obj_file}.")
    with open(obj_file, "w", encoding="utf-8") as f:


SUMMARY:
**Function Summary: `fix_docstring` Function**

### Purpose and Functionality

The `fix_docstring` function updates the docstrings of objects by replacing outdated parameter documentation with the latest one based on the object's signature. It ensures accuracy and consistency in documentations.

### Key Implementation Details

1.  **Inspecting Docstrings**:

    *   Reads source code from a given object.
    *   Identifies the lines where arguments are defined and replaced.
2.  **Matching Docstring with Signature**:

    *   Uses `match_docstring_with_signature` to determine the updated documentation based on the signature.

### Notable Dependencies

1  **External Dependency**:

    *   `re` for regular expression matching
2  **Inherited/Implemented Interfaces** (None mentioned in this function)

The code snippet fixes the docstrings of objects by replacing outdated parameter documentation with accurate ones. It uses regular expressions to identify lines where arguments are defined, reads source code from an object, and updates its content accordingly.


key: chroma:document
string_value:
CODE:

def check_docstrings(overwrite: bool = False, check_all: bool = False):
    """
    Check docstrings of all public objects that are callables and are documented. By default, only checks the diff.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether to fix inconsistencies or not.
        check_all (`bool`, *optional*, defaults to `False`):
            Whether to check all files.
    """
    module_diff_files = None
    if not check_all:
        module_diff_files = set()
        repo = Repo(PATH_TO_REPO)
        # Diff from index to unstaged files
        for modified_file_diff in repo.index.diff(None):
            if modified_file_diff.a_path.startswith("src/transformers"):
                module_diff_files.add(modified_file_diff.a_path)
        # Diff from index to `main`
        for modified_file_diff in repo.index.diff(repo.refs.main.commit):
            if modified_file_diff.a_path.startswith("src/transformers"):
                module_diff_files.add(modified_file_diff.a_path)
        # quick escape route: if there are no module files in the diff, skip this check
        if len(module_diff_files) == 0:
            return
        print("    Checking docstrings in the following files:" + "\n    - " + "\n    - ".join(module_diff_files))

    failures = []
    hard_failures = []
    to_clean = []
    for name in dir(transformers):
        # Skip objects that are private or not documented.
        if name.startswith("_") or ignore_undocumented(name) or name in OBJECTS_TO_IGNORE:
            continue

        obj = getattr(transformers, name)
        if not callable(obj) or not isinstance(obj, type) or getattr(obj, "__doc__", None) is None:
            continue

        # If we are checking against the diff, we skip objects that are not part of the diff.
        if module_diff_files is not None:
            object_file = find_source_file(getattr(transformers, name))
            object_file_relative_path = "src/" + str(object_file).split("/src/")[1]
            if object_file_relative_path not in module_diff_files:
                continue

        # Check docstring
        try:
            result = match_docstring_with_signature(obj)
            if result is not None:
                old_doc, new_doc = result
            else:
                old_doc, new_doc = None, None
        except Exception as e:
            print(e)
            hard_failures.append(name)
            continue
        if old_doc != new_doc:
            if overwrite:
                fix_docstring(obj, old_doc, new_doc)
            else:
                failures.append(name)
        elif not overwrite and new_doc is not None and ("<fill_type>" in new_doc or "<fill_docstring>" in new_doc):
            to_clean.append(name)

    # Deal with errors
    error_message = ""
    if len(hard_failures) > 0:
        error_message += (
            "The argument part of the docstrings of the following objects could not be processed, check they are "
            "properly formatted."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in hard_failures])
    if len(failures) > 0:
        error_message += (
            "The following objects docstrings do not match their signature. Run `make fix-copies` to fix this. "
            "In some cases, this error may be raised incorrectly by the docstring checker. If you think this is the "
            "case, you can manually check the docstrings and then add the object name to `OBJECTS_TO_IGNORE` in "
            "`utils/check_docstrings.py`."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in failures])
    if len(to_clean) > 0:
        error_message += (
            "The following objects docstrings contain templates you need to fix: search for `<fill_type>` or "
            "`<fill_docstring>`."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in to_clean])

    if len(error_message) > 0:
        error_message = "There was at least one problem when checking docstrings of public objects.\n" + error_message


SUMMARY:
**Summary**

The `check_docstrings` function checks docstrings of public, callable objects in the `transformers` module and its dependencies. It:

1. **Analyze Documentation**: Checks if existing docs match defined signatures.

2. **Fix Potential Issues**: Fixes inconsistencies or warnings reported by the tool.

3. **Report Potential Bugs**: Prints error messages about unmatched signatures, incomplete methods, and potential bugs.

**Key Functionality**

*   `check_all`: Optional flag to check all files.
*   `overwrite`: Optional flag to fix inconsistencies instead of logging failures.

**Implementation Details**

*   Verifies functions match expected signature patterns.
*   Skips private objects or those without docstrings.
*   Raises `ValueError` with error messages when reports detection issues.


key: chroma:document
string_value:
CODE:

def download_and_extract(task, data_dir):
    print(f"Downloading and extracting {task}...")
    data_file = f"{task}.zip"
    urllib.request.urlretrieve(TASK2PATH[task], data_file)
    with zipfile.ZipFile(data_file) as zip_ref:
        zip_ref.extractall(data_dir)
    os.remove(data_file)


SUMMARY:
**Summary:**

`download_and_extract` function downloads and extracts a zip file for a given task, extracting the contents to a specified data directory.

**Implementation Details:**

* Downloads from `taskId`
* Extracts contents using `zipfile.ZipFile.extractall()`
* Deletes temp zip file after extraction (`os.remove(data_file)`)

**Notable Dependencies:**

* `urllib.request.urlretrieve()` for downloading
* `zipfile` module for extracting zip file
* `os` module for deleting temporary files

**Primary Class/Function Names:**

* `task`
* `data_dir`
* `TASK2PATH[task]`: mapping of tasks to download paths


key: chroma:document
string_value:
CODE:

def format_mrpc(data_dir, path_to_data):
    print("Processing MRPC...")
    mrpc_dir = os.path.join(data_dir, "MRPC")
    if not os.path.isdir(mrpc_dir):
        os.mkdir(mrpc_dir)
    if path_to_data:
        mrpc_train_file = os.path.join(path_to_data, "msr_paraphrase_train.txt")
        mrpc_test_file = os.path.join(path_to_data, "msr_paraphrase_test.txt")
    else:
        print("Local MRPC data not specified, downloading data from %s" % MRPC_TRAIN)
        mrpc_train_file = os.path.join(mrpc_dir, "msr_paraphrase_train.txt")
        mrpc_test_file = os.path.join(mrpc_dir, "msr_paraphrase_test.txt")
        urllib.request.urlretrieve(MRPC_TRAIN, mrpc_train_file)
        urllib.request.urlretrieve(MRPC_TEST, mrpc_test_file)
    if not os.path.isfile(mrpc_train_file):
        raise ValueError(f"Train data not found at {mrpc_train_file}")
    if not os.path.isfile(mrpc_test_file):
        raise ValueError(f"Test data not found at {mrpc_test_file}")
    urllib.request.urlretrieve(TASK2PATH["MRPC"], os.path.join(mrpc_dir, "dev_ids.tsv"))

    dev_ids = []
    with open(os.path.join(mrpc_dir, "dev_ids.tsv"), encoding="utf8") as ids_fh:
        for row in ids_fh:
            dev_ids.append(row.strip().split("\t"))

    with open(mrpc_train_file, encoding="utf8") as data_fh, open(
        os.path.join(mrpc_dir, "train.tsv"), "w", encoding="utf8"
    ) as train_fh, open(os.path.join(mrpc_dir, "dev.tsv"), "w", encoding="utf8") as dev_fh:
        header = data_fh.readline()
        train_fh.write(header)
        dev_fh.write(header)
        for row in data_fh:
            label, id1, id2, s1, s2 = row.strip().split("\t")
            if [id1, id2] in dev_ids:
                dev_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1, s2))
            else:
                train_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1, s2))

    with open(mrpc_test_file, encoding="utf8") as data_fh, open(
        os.path.join(mrpc_dir, "test.tsv"), "w", encoding="utf8"
    ) as test_fh:
        header = data_fh.readline()
        test_fh.write("index\t#1 ID\t#2 ID\t#1 String\t#2 String\n")
        for idx, row in enumerate(data_fh):
            label, id1, id2, s1, s2 = row.strip().split("\t")
            test_fh.write("%d\t%s\t%s\t%s\t%s\n" % (idx, id1, id2, s1, s2))


SUMMARY:
**Function Summary: `format_mrpc`**

**Main Purpose and Functionality**
The `format_mrpc` function processes the MRPC (Microsoft Research Paraphrase Corpus) dataset by:

* Creating a directory for the dataset if it does not exist
* Downloading dataset files from a specified URL or using local data if specified
* Validating the presence of required files
* Preprocessing data into training, development, and testing sets

**Key Implementation Details**

* Uses `os.path.join` to construct file paths
* Utilizes `urllib.request.urlretrieve` to download external datasets
* Reads and writes data from/to files using `open` function with specified encoding
* Splits input file rows into individual labels, IDs, and strings

**Notable Dependencies and Requirements**

* Uses `os` module for file system operations
* Utilizes `urllib.request` module for downloading external datasets
* References classes and functions by exact name:
	+ `MRPC_TRAIN`
	+ `TASK2PATH["MRPC"]`

**External Dependencies:**

* `urllib.request`
* `os`
* Python's built-in file I/O operations


key: chroma:document
string_value:
CODE:

def download_diagnostic(data_dir):
    print("Downloading and extracting diagnostic...")
    if not os.path.isdir(os.path.join(data_dir, "diagnostic")):
        os.mkdir(os.path.join(data_dir, "diagnostic"))
    data_file = os.path.join(data_dir, "diagnostic", "diagnostic.tsv")
    urllib.request.urlretrieve(TASK2PATH["diagnostic"], data_file)
    print("\tCompleted!")


SUMMARY:
**Summary:**

The `download_diagnostic` function downloads and extracts diagnostic data from a specified directory. It:

* Creates the 'diagnostic' subdirectory if it does not exist
* Downloads the diagnostic file using `urlretrieve` from the `TASK2PATH["diagnostic"]` location
* Prints confirmation messages after completion

**Key details:**

* Uses `os.path.join` to construct paths for directory and file creation/export
* Implements `urllib.request.urlretrieve` for file download
* Returns no value (indicating a side-effecting function)

**Notable dependencies/requirements:**

* `os.path`
* `urllib.request`
* `TASK2PATH` (dictionary containing path to diagnostic data)


key: chroma:document
string_value:
CODE:

def get_tasks(task_names):
    task_names = task_names.split(",")
    if "all" in task_names:
        tasks = TASKS
    else:
        tasks = []
        for task_name in task_names:
            if task_name not in TASKS:
                raise ValueError(f"Task {task_name} not found!")
            tasks.append(task_name)


SUMMARY:
**Function Summary: `get_tasks`**

The `get_tasks` function retrieves the specified task names from a predefined list of tasks (`TASKS`). If "all" is among the input task names, it returns all available tasks. Otherwise, it checks each task name against the `TASKS` list and appends valid task names to a result list, raising an error for invalid task names.

**Key Implementation Details:**

* Uses string splitting to parse input task names.
* Checks if "all" is present in the input task names.

**Notable Dependencies or Requirements:**

* The `TASKS` list of predefined task names (assuming it's defined elsewhere).
* None explicitly listed, but implicit dependencies on Python's built-in data structures and error handling mechanisms.


key: chroma:document
string_value:
CODE:

def main(arguments):
    parser = argparse.ArgumentParser()
    parser.add_argument("--data_dir", help="directory to save data to", type=str, default="glue_data")
    parser.add_argument(
        "--tasks", help="tasks to download data for as a comma separated string", type=str, default="all"
    )
    parser.add_argument(
        "--path_to_mrpc",
        help="path to directory containing extracted MRPC data, msr_paraphrase_train.txt and msr_paraphrase_text.txt",
        type=str,
        default="",
    )
    args = parser.parse_args(arguments)

    if not os.path.isdir(args.data_dir):
        os.mkdir(args.data_dir)
    tasks = get_tasks(args.tasks)

    for task in tasks:
        if task == "MRPC":
            format_mrpc(args.data_dir, args.path_to_mrpc)
        elif task == "diagnostic":
            download_diagnostic(args.data_dir)
        else:


SUMMARY:
Here is a concise summary of the provided code chunk:

**Main Functionality:**
The `main` function orchestrates data preparation for multiple natural language processing (NLP) tasks. It:

1. Parses command-line arguments for task specifics (`--data_dir`, `--tasks`, and `--path_to_mrpc`) to download and process data.
2. Creates a directory to store the downloaded data if it does not exist.
3. Determines the tasks to be processed based on the comma-separated string provided in `--tasks`.
4. For each task, performs specific actions:
   - MRPC task: Formats MRPC data using `format_mrpc`.
   - Diagnostic task: Downloads diagnostic data using `download_diagnostic`.
   - Other tasks: Downloads and extracts data using `download_and_extract`.

**Key Implementation Details:**

* Utilizes `argparse` for command-line argument parsing.
* Uses the `get_tasks` function to determine the specific tasks to be processed.
* Implements a conditional statement to perform task-specific actions based on the parsed arguments.

**Notable Dependencies or Requirements:**

* `os` module for directory manipulation (e.g., creating directories).
* `argparse` library for command-line argument parsing.
* External dependencies:
	+ `get_tasks` function (not shown in this snippet).
	+ `download_diagnostic`, `format_mrpc`, and `download_and_extract` functions (not shown in this snippet).


key: chroma:document
string_value:
CODE:

def camel_case_split(identifier: str) -> List[str]:
    """
    Split a camel-cased name into words.

    Args:
        identifier (`str`): The camel-cased name to parse.

    Returns:
        `List[str]`: The list of words in the identifier (as seprated by capital letters).

    Example:

    ```py
    >>> camel_case_split("CamelCasedClass")
    ["Camel", "Cased", "Class"]
    ```
    """
    # Regex thanks to https://stackoverflow.com/questions/29916065/how-to-do-camelcase-split-in-python
    matches = re.finditer(".+?(?:(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|$)", identifier)


SUMMARY:
** camel_case_split() Function Summary **


*   **Purpose and Functionality:** 
    Splits a camel-cased string into individual words based on capitalization.

        *   Type Hint: `identifier (str) -> List[str]`
        *   Returns: A list of words separated by capital letters.

*   **Implementation Details:** 
    - Uses regular expressions with look-behind and look-ahead assertions to match word boundaries.
    - Utilizes the `re.finditer` function to extract matches from the input string.


*   **Notable Dependencies or Requirements:**
    - `re` module for regex functionality
    - No external dependencies


        Example usage:

            ```py
>>> camel_case_split("CamelCasedClass")
['Camel', 'Cased', 'Class']
```


key: chroma:document
string_value:
CODE:

def get_frameworks_table() -> pd.DataFrame:
    """
    Generates a dataframe containing the supported auto classes for each model type, using the content of the auto
    modules.
    """
    # Dictionary model names to config.
    config_maping_names = transformers_module.models.auto.configuration_auto.CONFIG_MAPPING_NAMES
    model_prefix_to_model_type = {
        config.replace("Config", ""): model_type for model_type, config in config_maping_names.items()
    }

    # Dictionaries flagging if each model prefix has a backend in PT/TF/Flax.
    pt_models = collections.defaultdict(bool)
    tf_models = collections.defaultdict(bool)
    flax_models = collections.defaultdict(bool)

    # Let's lookup through all transformers object (once) and find if models are supported by a given backend.
    for attr_name in dir(transformers_module):
        lookup_dict = None
        if _re_tf_models.match(attr_name) is not None:
            lookup_dict = tf_models
            attr_name = _re_tf_models.match(attr_name).groups()[0]
        elif _re_flax_models.match(attr_name) is not None:
            lookup_dict = flax_models
            attr_name = _re_flax_models.match(attr_name).groups()[0]
        elif _re_pt_models.match(attr_name) is not None:
            lookup_dict = pt_models
            attr_name = _re_pt_models.match(attr_name).groups()[0]

        if lookup_dict is not None:
            while len(attr_name) > 0:
                if attr_name in model_prefix_to_model_type:
                    lookup_dict[model_prefix_to_model_type[attr_name]] = True
                    break
                # Try again after removing the last word in the name
                attr_name = "".join(camel_case_split(attr_name)[:-1])

    all_models = set(list(pt_models.keys()) + list(tf_models.keys()) + list(flax_models.keys()))
    all_models = list(all_models)
    all_models.sort()

    data = {"model_type": all_models}
    data["pytorch"] = [pt_models[t] for t in all_models]
    data["tensorflow"] = [tf_models[t] for t in all_models]
    data["flax"] = [flax_models[t] for t in all_models]

    # Now let's find the right processing class for each model. In order we check if there is a Processor, then a
    # Tokenizer, then a FeatureExtractor, then an ImageProcessor
    processors = {}
    for t in all_models:
        if t in transformers_module.models.auto.processing_auto.PROCESSOR_MAPPING_NAMES:
            processors[t] = "AutoProcessor"
        elif t in transformers_module.models.auto.tokenization_auto.TOKENIZER_MAPPING_NAMES:
            processors[t] = "AutoTokenizer"
        elif t in transformers_module.models.auto.image_processing_auto.IMAGE_PROCESSOR_MAPPING_NAMES:
            processors[t] = "AutoImageProcessor"
        elif t in transformers_module.models.auto.feature_extraction_auto.FEATURE_EXTRACTOR_MAPPING_NAMES:
            processors[t] = "AutoFeatureExtractor"
        else:
            # Default to AutoTokenizer if a model has nothing, for backward compatibility.
            processors[t] = "AutoTokenizer"

    data["processor"] = [processors[t] for t in all_models]



SUMMARY:
**Function Summary: `get_frameworks_table`**

**Primary Purpose:** Generates a pandas DataFrame containing the supported auto classes for each model type, using the content of the auto modules.

**Key Implementation Details:**

* Uses two dictionaries to map model prefixes to model types (`model_prefix_to_model_type`)
* Populates three separate dictionaries (`pt_models`, `tf_models`, and `flax_models`) to track backend support for each model prefix
* Iterates through all transformers object attributes using regular expressions to identify models supported by PT, TF, or Flax backends

**Notable Dependencies and Requirements:**

* `pd` (pandas library) for data manipulation
* `transformers_module` (external dependencies) containing configuration mapping names and processing class mappings
* `_re_tf_models`, `_re_flax_models`, and `_re_pt_models` (regex patterns used to identify models supported by PT, TF, or Flax backends)
* `collections` (utility module) for using defaultdicts


key: chroma:document
string_value:
CODE:

def update_pipeline_and_auto_class_table(table: Dict[str, Tuple[str, str]]) -> Dict[str, Tuple[str, str]]:
    """
    Update the table maping models to pipelines and auto classes without removing old keys if they don't exist anymore.

    Args:
        table (`Dict[str, Tuple[str, str]]`):
            The existing table mapping model names to a tuple containing the pipeline tag and the auto-class name with
            which they should be used.

    Returns:
        `Dict[str, Tuple[str, str]]`: The updated table in the same format.
    """
    auto_modules = [
        transformers_module.models.auto.modeling_auto,
        transformers_module.models.auto.modeling_tf_auto,
        transformers_module.models.auto.modeling_flax_auto,
    ]
    for pipeline_tag, model_mapping, auto_class in PIPELINE_TAGS_AND_AUTO_MODELS:
        model_mappings = [model_mapping, f"TF_{model_mapping}", f"FLAX_{model_mapping}"]
        auto_classes = [auto_class, f"TF_{auto_class}", f"Flax_{auto_class}"]
        # Loop through all three frameworks
        for module, cls, mapping in zip(auto_modules, auto_classes, model_mappings):
            # The type of pipeline may not exist in this framework
            if not hasattr(module, mapping):
                continue
            # First extract all model_names
            model_names = []
            for name in getattr(module, mapping).values():
                if isinstance(name, str):
                    model_names.append(name)
                else:
                    model_names.extend(list(name))

            # Add pipeline tag and auto model class for those models
            table.update({model_name: (pipeline_tag, cls) for model_name in model_names})



SUMMARY:
**Function Summary**

### `update_pipeline_and_auto_class_table`

Updates the table mapping models to pipelines and auto classes without removing old keys.

#### Parameters
- `table`: The existing table mapping model names to a tuple containing the pipeline tag and the auto-class name (Dict[str, Tuple[str, str]])

#### Returns
- `Dict[str, Tuple[str, str]]`: The updated table in the same format

#### Implementation Details

- Updates table with pipeline tags and auto-model classes for each framework (transformers_module.models.auto.modeling_auto, transformers_module.models.auto.modeling_tf_auto, transformers_module.models.auto.modeling_flax_auto)
- Handles cases where pipeline type may not exist in a given framework
- Uses model mappings and pipepline tags to populate the table

#### Notable Dependencies
- `PIPELINE_TAGS_AND_AUTO_MODELS`
- `transformers_module.models.auto.modeling_auto`, `transformers_module.models.auto.modeling_tf_auto`, `transformers_module.models.auto.modeling_flax_auto`

#### External Dependencies
- `transformers_module`

#### Notes

This function iterates over a list of frameworks, extracts model names, and updates the table with pipeline tags and auto-model classes.


key: chroma:document
string_value:
CODE:

def update_metadata(token: str, commit_sha: str):
    """
    Update the metadata for the Transformers repo in `huggingface/transformers-metadata`.

    Args:
        token (`str`): A valid token giving write access to `huggingface/transformers-metadata`.
        commit_sha (`str`): The commit SHA on Transformers corresponding to this update.
    """
    frameworks_table = get_frameworks_table()
    frameworks_dataset = Dataset.from_pandas(frameworks_table)

    resolved_tags_file = hf_hub_download(
        "huggingface/transformers-metadata", "pipeline_tags.json", repo_type="dataset", token=token
    )
    tags_dataset = Dataset.from_json(resolved_tags_file)
    table = {
        tags_dataset[i]["model_class"]: (tags_dataset[i]["pipeline_tag"], tags_dataset[i]["auto_class"])
        for i in range(len(tags_dataset))
    }
    table = update_pipeline_and_auto_class_table(table)

    # Sort the model classes to avoid some nondeterministic updates to create false update commits.
    model_classes = sorted(table.keys())
    tags_table = pd.DataFrame(
        {
            "model_class": model_classes,
            "pipeline_tag": [table[m][0] for m in model_classes],
            "auto_class": [table[m][1] for m in model_classes],
        }
    )
    tags_dataset = Dataset.from_pandas(tags_table)

    hub_frameworks_json = hf_hub_download(
        repo_id="huggingface/transformers-metadata",
        filename="frameworks.json",
        repo_type="dataset",
        token=token,
    )
    with open(hub_frameworks_json) as f:
        hub_frameworks_json = f.read()

    hub_pipeline_tags_json = hf_hub_download(
        repo_id="huggingface/transformers-metadata",
        filename="pipeline_tags.json",
        repo_type="dataset",
        token=token,
    )
    with open(hub_pipeline_tags_json) as f:
        hub_pipeline_tags_json = f.read()

    with tempfile.TemporaryDirectory() as tmp_dir:
        frameworks_dataset.to_json(os.path.join(tmp_dir, "frameworks.json"))
        tags_dataset.to_json(os.path.join(tmp_dir, "pipeline_tags.json"))

        with open(os.path.join(tmp_dir, "frameworks.json")) as f:
            frameworks_json = f.read()
        with open(os.path.join(tmp_dir, "pipeline_tags.json")) as f:
            pipeline_tags_json = f.read()

        frameworks_equal = hub_frameworks_json == frameworks_json
        hub_pipeline_tags_equal = hub_pipeline_tags_json == pipeline_tags_json

        if frameworks_equal and hub_pipeline_tags_equal:
            print("No updates on the Hub, not pushing the metadata files.")
            return

        if commit_sha is not None:
            commit_message = (
                f"Update with commit {commit_sha}\n\nSee: "
                f"https://github.com/huggingface/transformers/commit/{commit_sha}"
            )
        else:
            commit_message = "Update"

        upload_folder(
            repo_id="huggingface/transformers-metadata",
            folder_path=tmp_dir,
            repo_type="dataset",
            token=token,
            commit_message=commit_message,


SUMMARY:
**Function Summary:**

`update_metadata`

**Purpose:** Updates the metadata for the Transformers repository by processing and uploading dataset files.

**Key Implementation Details:**

* Downloads pipeline_tags.json and frameworks.json dataset files from huggingface/transformers-metadata
* Processes framework and tag data from downloaded files
* Checks if updates need to be pushed to GitHub and commits message is generated accordingly

**Noteable Dependencies/Requirements:**

* `get_frameworks_table` function (not shown in this code snippet)
* `Dataset.from_json` and `Dataset.from_pandas` functions (from the Hugging Face datasets library)
* `hf_hub_download` and `upload_folder` functions (from the Hugging Face Hub client library)

**Interfaces/ Method Signatures:**

* `update_pipeline_and_auto_class_table(table)`: Updates table with processed framework data
* `get_frameworks_table()`: Returns a pandas dataframe containing frameworks data

**External Dependencies:**

* `huggingface/transformers-metadata`
* `huggingfacedatasets` library
* `huggingface_hub` library


key: chroma:document
string_value:
CODE:

def check_pipeline_tags():
    """
    Check all pipeline tags are properly defined in the `PIPELINE_TAGS_AND_AUTO_MODELS` constant of this script.
    """
    in_table = {tag: cls for tag, _, cls in PIPELINE_TAGS_AND_AUTO_MODELS}
    pipeline_tasks = transformers_module.pipelines.SUPPORTED_TASKS
    missing = []
    for key in pipeline_tasks:
        if key not in in_table:
            model = pipeline_tasks[key]["pt"]
            if isinstance(model, (list, tuple)):
                model = model[0]
            model = model.__name__
            if model not in in_table.values():
                missing.append(key)

    if len(missing) > 0:
        msg = ", ".join(missing)
        raise ValueError(
            "The following pipeline tags are not present in the `PIPELINE_TAGS_AND_AUTO_MODELS` constant inside "
            f"`utils/update_metadata.py`: {msg}. Please add them!"


SUMMARY:
**Summary:**

This function, `check_pipeline_tags()`, detects and raises an error if any pipeline tasks are missing from the predefined `PIPELINE_TAGS_AND_AUTO_MODELS` constant.

**Key Details:**

- Verifies that all task pipelines match the predefined auto-models.
- Checks against `transformers_module.pipelines.SUPPORTED_TASKS`.
- Raises a `ValueError` with missing task pipelines as input when not found in the defined constants.

**Implementation Notes:**

* Utilizes dictionary comprehension to create an in-table mapping of pipeline tags and models.
* Iterates through supported tasks, checking their existence in the predefined constant map (`in_table`).
* Raises an error if the `missing` list is populated during iteration. 

This function ensures that all defined pipelines have corresponding auto-models.


key: docstring
string_value:
r"""
The argument `test_file` in this file refers to a model test file. This should be a string of the from
`tests/models/*/test_modeling_*.py`.
"""


key: chroma:document
string_value:
CODE:

def get_module_path(test_file):
    """Return the module path of a model test file."""
    components = test_file.split(os.path.sep)
    if components[0:2] != ["tests", "models"]:
        raise ValueError(
            "`test_file` should start with `tests/models/` (with `/` being the OS specific path separator). Got "
            f"{test_file} instead."
        )
    test_fn = components[-1]
    if not test_fn.endswith("py"):
        raise ValueError(f"`test_file` should be a python file. Got {test_fn} instead.")
    if not test_fn.startswith("test_modeling_"):
        raise ValueError(
            f"`test_file` should point to a file name of the form `test_modeling_*.py`. Got {test_fn} instead."
        )

    components = components[:-1] + [test_fn.replace(".py", "")]
    test_module_path = ".".join(components)



SUMMARY:
**Function Summary**

The `get_module_path` function takes a `test_file` path as input and returns the corresponding module path. It validates that the input path matches the expected format `tests/modeling/*.py`, raising errors for any deviations.

**Key Implementation Details**

- Splits the input `test_file` into components using the OS-specific path separator.
- Verifies that the first two components are "tests" and "models," and the last component ends with ".py."
- Reconstructs the module path from the validated components.

**Notable Dependencies and Requirements**

- Uses the `os` module for handling file paths.
- References `python files` directly in validation checks.


key: chroma:document
string_value:
CODE:

def get_test_module(test_file):
    """Get the module of a model test file."""
    test_module_path = get_module_path(test_file)
    try:
        test_module = importlib.import_module(test_module_path)
    except AttributeError as exc:
        # e.g. if you have a `tests` folder in `site-packages`, created by another package, when trying to import
        # `tests.models...`
        raise ValueError(
            f"Could not import module {test_module_path}. Confirm that you don't have a package with the same root "
            "name installed or in your environment's `site-packages`."
        ) from exc



SUMMARY:
**Summary:**

This function, `get_test_module`, retrieves and loads the Python module corresponding to a given test file by its path. It identifies if the import is successful or not and provides an informative error message if an AttributeError occurs.

**Key Implementation Details:**

* Imports a module using `importlib.import_module` with a specified path.
* Raises a ValueError with an informative message if the import fails due to a common issue.

**Notable Dependencies:**

* `importlib`
* None

**Interfaces/Classes:**

* No external interface or class is used in this function.


key: chroma:document
string_value:
CODE:

def get_tester_classes(test_file):
    """Get all classes in a model test file whose names ends with `ModelTester`."""
    tester_classes = []
    test_module = get_test_module(test_file)
    for attr in dir(test_module):
        if attr.endswith("ModelTester"):
            tester_classes.append(getattr(test_module, attr))

    # sort with class names


SUMMARY:
**Code Summary**

### Purpose and Functionality

Get all classes in a model test file whose names end with "ModelTester".

### Implementation Details

* Uses the `dir()` function to list attributes of the `test_module` object.
* Filters attributes based on name ending with "ModelTester".
* Retrieves attribute objects using `getattr()`.
* Sorts retrieved classes by their names using the `sorted()` function and a lambda function.

### Notable Dependencies or Requirements

* `get_test_module`: Returns the test module for the given test file (not shown in this code snippet).
 
This summary preserves detailed information from the original code while minimizing unnecessary details.


key: chroma:document
string_value:
CODE:

def get_test_classes(test_file):
    """Get all [test] classes in a model test file with attribute `all_model_classes` that are non-empty.

    These are usually the (model) test classes containing the (non-slow) tests to run and are subclasses of one of the
    classes `ModelTesterMixin`, `TFModelTesterMixin` or `FlaxModelTesterMixin`, as well as a subclass of
    `unittest.TestCase`. Exceptions include `RagTestMixin` (and its subclasses).
    """
    test_classes = []
    test_module = get_test_module(test_file)
    for attr in dir(test_module):
        attr_value = getattr(test_module, attr)
        # (TF/Flax)ModelTesterMixin is also an attribute in specific model test module. Let's exclude them by checking
        # `all_model_classes` is not empty (which also excludes other special classes).
        model_classes = getattr(attr_value, "all_model_classes", [])
        if len(model_classes) > 0:
            test_classes.append(attr_value)

    # sort with class names


SUMMARY:
Here is a concise summary of the code chunk:

**Function Purpose**
The `get_test_classes` function retrieves and sorts a list of non-empty test classes in a model test file that inherit from specific mixins (`ModelTesterMixin`, `TFModelTesterMixin`, or `FlaxModelTesterMixin`) or `unittest.TestCase`.

**Key Implementation Details**

* Iterates through the module's attributes to find classes with an empty `all_model_classes` attribute.
* Excludes exceptions like `RagTestMixin` and its subclasses.
* Sorts the test classes by their names using `__name__` attribute.

**Notable Dependencies and Requirements**

* The `get_test_module` function is called from within this function, suggesting a dependency on another module (`test_file`).
* This function relies on `unittest.TestCase`, indicating a requirement for testing framework.


key: chroma:document
string_value:
CODE:

def get_model_classes(test_file):
    """Get all model classes that appear in `all_model_classes` attributes in a model test file."""
    test_classes = get_test_classes(test_file)
    model_classes = set()
    for test_class in test_classes:
        model_classes.update(test_class.all_model_classes)

    # sort with class names


SUMMARY:
**Function Summary**

`get_model_classes`

Returns a sorted list of unique model classes from `all_model_classes` attributes in a model test file.

**Key Implementation Details**

* Uses `set` to eliminate duplicate class instances
* Iterates over `test_classes`, updating `model_classes` with `all_model_classes`
* Sorts classes by name using a case-insensitive comparison

**Notable Dependencies and Requirements**

* `get_test_classes` function is called with `test_file` as an argument
* Uses `__name__` attribute of class names for sorting comparison
* No external dependencies are listed due to lack of information


key: chroma:document
string_value:
CODE:

def get_model_tester_from_test_class(test_class):
    """Get the model tester class of a model test class."""
    test = test_class()
    if hasattr(test, "setUp"):
        test.setUp()

    model_tester = None
    if hasattr(test, "model_tester"):
        # `(TF/Flax)ModelTesterMixin` has this attribute default to `None`. Let's skip this case.
        if test.model_tester is not None:
            model_tester = test.model_tester.__class__



SUMMARY:
**Summary**

The `get_model_tester_from_test_class` function takes a `test_class` as input and returns the model tester class associated with it. It checks for the presence of an instance with an explicit `setUp` method, then attempts to retrieve the model tester class from the instance's attributes.

**Key Implementation Details**

* Tries to call `setUp` on the test instance if present
* Ignores cases where `model_tester` is set to `None`, likely due to default values in `ModelTesterMixin`

**Notable Dependencies/Requirements**

* `test_class`: The input class being analyzed
* `TF/Flax` (implied): The framework used for the `ModelTesterMixin`
* No external dependencies noted

**Notes**: This code appears to be part of a testing harness, likely for machine learning models.


key: chroma:document
string_value:
CODE:

def get_test_classes_for_model(test_file, model_class):
    """Get all [test] classes in `test_file` that have `model_class` in their `all_model_classes`."""
    test_classes = get_test_classes(test_file)

    target_test_classes = []
    for test_class in test_classes:
        if model_class in test_class.all_model_classes:
            target_test_classes.append(test_class)

    # sort with class names


SUMMARY:
**Function Summary**

`get_test_classes_for_model`: Returns a list of test classes from `test_file` that include `model_class`. 

**Key Implementation Details**

* Uses `sorted` function for alphabetical ordering by class name (`x.__name__`)
* Iterates through all test classes using `get_test_classes(test_file)`
* Filters test classes based on presence of `model_class` in `all_model_classes`

**Input/Output**

* `test_file`: input file containing test classes
* `model_class`: model class to search for in test classes
* Returns a sorted list of matching test classes

**Dependencies/Requirements**

* `get_test_classes` function from same library or module


key: chroma:document
string_value:
CODE:

def get_tester_classes_for_model(test_file, model_class):
    """Get all model tester classes in `test_file` that are associated to `model_class`."""
    test_classes = get_test_classes_for_model(test_file, model_class)

    tester_classes = []
    for test_class in test_classes:
        tester_class = get_model_tester_from_test_class(test_class)
        if tester_class is not None:
            tester_classes.append(tester_class)

    # sort with class names


SUMMARY:
**Summary**

Returns a list of model tester classes from a test file (`test_file`) associated with the specified `model_class`, sorted alphabetically by class name.

**Key Details**

* **Functionality**: Retrieves and filters model tester classes based on the provided `test_file` and `model_class`.
* **Algorithm/Design Choice**: Sorts filtered tester classes using the `__name__` attribute.
* **Method Signatures**:
	+ `get_test_classes_for_model()`: returns list of test classes for a given model
	+ `get_model_tester_from_test_class(test_class)`: returns associated model tester class from a test class

**Dependencies**

* `get_test_classes_for_model`
* `get_model_tester_from_test_class`


key: chroma:document
string_value:
CODE:

def get_test_to_tester_mapping(test_file):
    """Get a mapping from [test] classes to model tester classes in `test_file`.

    This uses `get_test_classes` which may return classes that are NOT subclasses of `unittest.TestCase`.
    """
    test_classes = get_test_classes(test_file)
    test_tester_mapping = {test_class: get_model_tester_from_test_class(test_class) for test_class in test_classes}


SUMMARY:
**Summary**

`get_test_to_tester_mapping` function:
 
* Maps test classes to model tester classes based on inheritance from `unittest.TestCase`.
* Uses `get_test_classes` to retrieve test class list and `get_model_tester_from_test_class` to map each test class.

**Key Details:**

* Reusable in a unit testing framework.
* May return classes that are not subclasses of `unittest.TestCase`.

**Implementation Notes:**

* Uses dictionary comprehension for efficient creation of mapping.
* Relies on `get_test_classes` and `get_model_tester_from_test_class` external functions.


key: chroma:document
string_value:
CODE:

def get_model_to_test_mapping(test_file):
    """Get a mapping from model classes to test classes in `test_file`."""
    model_classes = get_model_classes(test_file)
    model_test_mapping = {
        model_class: get_test_classes_for_model(test_file, model_class) for model_class in model_classes
    }


SUMMARY:
**Summary:**

`get_model_to_test_mapping` function generates a dictionary mapping `model_classes` to their corresponding test classes (`test_classes`) based on the input `test_file`. It uses `get_model_classes` and `get_test_classes_for_model` functions.

**Key Details:**

- Uses `get_model_classes(test_file)` to retrieve model class names
- Populates a dictionary (`model_test_mapping`) with models as keys and test classes as values using map comprehension

**Notable Dependencies/Requirements:**

- `get_model_classes(test_file)`
- `get_test_classes_for_model(test_file, model_class)`

It can be summarized as:
```python
def get_model_to_test_mapping(test_file):
    # Maps model classes to their corresponding test classes
    model_classes = get_model_classes(test_file)
    model_test_mapping = {model_class: get_test_classes_for_model(test_file, model_class) for model_class in model_classes}
    return model_test_mapping
```


key: chroma:document
string_value:
CODE:

def get_model_to_tester_mapping(test_file):
    """Get a mapping from model classes to model tester classes in `test_file`."""
    model_classes = get_model_classes(test_file)
    model_to_tester_mapping = {
        model_class: get_tester_classes_for_model(test_file, model_class) for model_class in model_classes
    }


SUMMARY:
**Summary**

`get_model_to_tester_mapping` function maps model classes to their corresponding tester classes in a given test file.

**Key Details:**

* Uses `model_classes` from `get_model_classes` for iteration
* Iterates using dictionary comprehension to create mappings of model classes to tester classes
* Returns a dictionary `model_to_tester_mapping`

**Notable Dependencies and Interfaces:**

* References `get_model_classes` and `get_tester_classes_for_model` functions (not shown)


key: chroma:document
string_value:
CODE:

def to_json(o):
    """Make the information succinct and easy to read.

    Avoid the full class representation like `<class 'transformers.models.bert.modeling_bert.BertForMaskedLM'>` when
    displaying the results. Instead, we use class name (`BertForMaskedLM`) for the readability.
    """
    if isinstance(o, str):
        return o
    elif isinstance(o, type):
        return o.__name__
    elif isinstance(o, (list, tuple)):
        return [to_json(x) for x in o]
    elif isinstance(o, dict):
        return {to_json(k): to_json(v) for k, v in o.items()}
    else:


SUMMARY:
**Concise Summary**

### Function Purpose and Main Functionality
The `to_json` function converts an object `o` into a human-readable JSON format. It handles various data types by recursively processing nested structures, truncating class representations for better readability.

### Key Implementation Details

* Handles string, type alias, list, tuple, and dictionary conversions using recursive calls.
* Uses `__name__` attribute to display only class names instead of full class representations.

### Notable Dependencies or Requirements
None mentioned, but it assumes objects are usable in a context that supports nested data processing (e.g., lists, dictionaries).

### Key Method Signatures and Interfaces

* Implicit: No explicit interfaces defined.
* Explicit: None.


key: chroma:document
string_value:
CODE:

def get_checkpoint_from_config_class(config_class):
    checkpoint = None

    # source code of `config_class`
    config_source = inspect.getsource(config_class)
    checkpoints = _re_checkpoint.findall(config_source)

    # Each `checkpoint` is a tuple of a checkpoint name and a checkpoint link.
    # For example, `('google-bert/bert-base-uncased', 'https://huggingface.co/google-bert/bert-base-uncased')`
    for ckpt_name, ckpt_link in checkpoints:
        # allow the link to end with `/`
        if ckpt_link.endswith("/"):
            ckpt_link = ckpt_link[:-1]

        # verify the checkpoint name corresponds to the checkpoint link
        ckpt_link_from_name = f"https://huggingface.co/{ckpt_name}"
        if ckpt_link == ckpt_link_from_name:
            checkpoint = ckpt_name
            break



SUMMARY:
**Summary:**

`get_checkpoint_from_config_class` function retrieves the corresponding checkpoint name from a provided config class's source code.

*   **Purpose:** To extract the checkpoint name from a config class based on its implementation details.
*   **Key Details:**
    *   Uses regular expression to find checkpoints in the config class source code.
    *   Verifies each checkpoint by comparing its name with the corresponding link.
*   **Notable Dependencies/Requirements:**
    *   `_re_checkpoint` regular expression pattern
    *   `inspect.getsource(config_class)` function
*   **Interfaces/Methods:** None (uses built-in Python functions and modules)
*   **External Dependencies:**

    *   `_re_checkpoint`
    *   `config_class` (input parameter)

**Summary Focus Criteria Met:**

1. Precise technical terms used: "config class", "checkpoint name", "regular expression pattern"
2. Class/function/variable names preserved exactly.
3. Primary purpose stated: "To extract the checkpoint name from a config class based on its implementation details."
4. Key implementation details narrowed down to:
    *   Regular expression usage
    *   Comparison verification for checkpoints
5. Notable dependencies listed with exact class names and functions: `_re_checkpoint`, `config_class`.
6. No external modules or libraries required besides built-in Python functions.


key: chroma:document
string_value:
CODE:

def check_config_docstrings_have_checkpoints():
    configs_without_checkpoint = []

    for config_class in list(CONFIG_MAPPING.values()):
        # Skip deprecated models
        if "models.deprecated" in config_class.__module__:
            continue
        checkpoint = get_checkpoint_from_config_class(config_class)

        name = config_class.__name__
        if checkpoint is None and name not in CONFIG_CLASSES_TO_IGNORE_FOR_DOCSTRING_CHECKPOINT_CHECK:
            configs_without_checkpoint.append(name)

    if len(configs_without_checkpoint) > 0:
        message = "\n".join(sorted(configs_without_checkpoint))
        raise ValueError(
            f"The following configurations don't contain any valid checkpoint:\n{message}\n\n"
            "The requirement is to include a link pointing to one of the models of this architecture in the "
            "docstring of the config classes listed above. The link should have be a markdown format like "
            "[myorg/mymodel](https://huggingface.co/myorg/mymodel)."


SUMMARY:
**Function Summary**

`check_config_docstrings_have_checkpoints()`: Verifies if all configuration classes in `CONFIG_MAPPING` have valid checkpoints in their docstrings. Raises a ValueError if any configurations lack checkpoint information.

**Key Implementation Details**

* Skips deprecated models by checking for "models.deprecated" in the module name.
* Iterates over config classes in `CONFIG_MAPPING`, appending names of missing checkpoint info to `configs_without_checkpoint` list.
* If no checkpoints found, raises a ValueError with an error message and sorted list of configuration class names.

**Notable Dependencies and Requirements**

* `CONFIG_MAPPING`: mapping of configuration classes
* `CONFIG_CLASSES_TO_IGNORE_FOR_DOCSTRING_CHECKPOINT_CHECK`: list of ignored configuration class names for docstring checkpoint check
* `get_checkpoint_from_config_class(config_class)`: function to retrieve checkpoint from a config class
* `sorted()` function for sorting configuration class names.


key: chroma:document
string_value:
CODE:

def handle_test_results(test_results):
    expressions = test_results.split(" ")

    failed = 0
    success = 0

    # When the output is short enough, the output is surrounded by = signs: "== OUTPUT =="
    # When it is too long, those signs are not present.
    time_spent = expressions[-2] if "=" in expressions[-1] else expressions[-1]

    for i, expression in enumerate(expressions):
        if "failed" in expression:
            failed += int(expressions[i - 1])
        if "passed" in expression:
            success += int(expressions[i - 1])



SUMMARY:
**Summary**

The `handle_test_results` function parses test results to extract failure and success counts, as well as the timestamp. Key aspects include:

*   **Purpose**: Parse test output to calculate failure and success metrics.
*   **Implementation Details**:
    *   Splits test output into expressions using spaces as separators.
    *   Calculates `time_spent` by detecting presence of `== OUTPUT ==` signs in the last expression.
    *   Iterates through expressions, incrementing `failed` and `success` counts based on specific checks (e.g., "failed" or "passed").
*   **Notable Dependencies/Requirements**:
    *   None

**Detailed Information**

*   Method Signatures: Not applicable
*   Interfaces Implemented: None
*   External Dependencies: None


key: chroma:document
string_value:
CODE:

def extract_first_line_failure(failures_short_lines):
    failures = {}
    file = None
    in_error = False
    for line in failures_short_lines.split("\n"):
        if re.search(r"_ \[doctest\]", line):
            in_error = True
            file = line.split(" ")[2]
        elif in_error and not line.split(" ")[0].isdigit():
            failures[file] = line
            in_error = False



SUMMARY:
**Summary:**

This function, `extract_first_line_failure`, processes text data from "short lines" of failures. It categorizes them into file-related failures by identifying the `_ [doctest]` prefix and extracts specific information about the error.

**Key Details:**

- **Algorithm:** Uses regular expressions to scan short lines for a pattern (_ [doctest]).
- **Method Signatures/Interfaces:** Takes `failures_short_lines` as input and returns a dictionary of failures (`failures`) with file names.
- **Dependencies:**
  - Utilizes external `re` (regular expression) module.
  - No imported classes/functions besides built-in Python.

**Context:**
This function appears to be part of a code analysis or testing framework, likely used for data extraction from test output logs.


key: chroma:document
string_value:
CODE:

class Message:
    def __init__(self, title: str, doc_test_results: Dict):
        self.title = title

        self.n_success = sum(job_result["n_success"] for job_result in doc_test_results.values())
        self.n_failures = sum(job_result["n_failures"] for job_result in doc_test_results.values())
        self.n_tests = self.n_success + self.n_failures

        # Failures and success of the modeling tests
        self.doc_test_results = doc_test_results

    @property
    def time(self) -> str:
        all_results = [*self.doc_test_results.values()]
        time_spent = [r["time_spent"].split(", ")[0] for r in all_results if len(r["time_spent"])]
        total_secs = 0

        for time in time_spent:
            time_parts = time.split(":")

            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.
            if len(time_parts) == 1:
                time_parts = [0, 0, time_parts[0]]

            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])
            total_secs += hours * 3600 + minutes * 60 + seconds

        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60
        return f"{int(hours)}h{int(minutes)}m{int(seconds)}s"

    @property
    def header(self) -> Dict:
        return {"type": "header", "text": {"type": "plain_text", "text": self.title}}

    @property
    def no_failures(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": f"🌞 There were no failures: all {self.n_tests} tests passed. The suite ran in {self.time}.",
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def failures(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": (
                    f"There were {self.n_failures} failures, out of {self.n_tests} tests.\nThe suite ran in"
                    f" {self.time}."
                ),
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def category_failures(self) -> List[Dict]:
        failure_blocks = []

        MAX_ERROR_TEXT = 3000 - len("The following examples had failures:\n\n\n\n") - len("[Truncated]\n")
        line_length = 40
        category_failures = {k: v["failed"] for k, v in doc_test_results.items() if isinstance(v, dict)}

        def single_category_failures(category, failures):
            text = ""
            if len(failures) == 0:
                return ""
            text += f"*{category} failures*:".ljust(line_length // 2).rjust(line_length // 2) + "\n"

            for idx, failure in enumerate(failures):
                new_text = text + f"`{failure}`\n"
                if len(new_text) > MAX_ERROR_TEXT:
                    text = text + "[Truncated]\n"
                    break
                text = new_text

            return text

        for category, failures in category_failures.items():
            report = single_category_failures(category, failures)
            if len(report) == 0:
                continue
            block = {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": f"The following examples had failures:\n\n\n{report}\n",
                },
            }
            failure_blocks.append(block)

        return failure_blocks

    @property
    def payload(self) -> str:
        blocks = [self.header]

        if self.n_failures > 0:
            blocks.append(self.failures)

        if self.n_failures > 0:
            blocks.extend(self.category_failures)

        if self.n_failures == 0:
            blocks.append(self.no_failures)

        return json.dumps(blocks)

    @staticmethod
    def error_out():
        payload = [
            {
                "type": "section",
                "text": {
                    "type": "plain_text",
                    "text": "There was an issue running the tests.",
                },
                "accessory": {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                    "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
                },
            }
        ]

        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(payload)}))

        client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            text="There was an issue running the tests.",
            blocks=payload,
        )

    def post(self):
        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(self.payload)}))

        text = f"{self.n_failures} failures out of {self.n_tests} tests," if self.n_failures else "All tests passed."

        self.thread_ts = client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            blocks=self.payload,
            text=text,
        )

    def get_reply_blocks(self, job_name, job_link, failures, text):
        # `text` must be less than 3001 characters in Slack SDK
        # keep some room for adding "[Truncated]" when necessary
        MAX_ERROR_TEXT = 3000 - len("[Truncated]")

        failure_text = ""
        for key, value in failures.items():
            new_text = failure_text + f"*{key}*\n_{value}_\n\n"
            if len(new_text) > MAX_ERROR_TEXT:
                # `failure_text` here has length <= 3000
                failure_text = failure_text + "[Truncated]"
                break
            # `failure_text` here has length <= MAX_ERROR_TEXT
            failure_text = new_text

        title = job_name
        content = {"type": "section", "text": {"type": "mrkdwn", "text": text}}

        if job_link is not None:
            content["accessory"] = {
                "type": "button",
                "text": {"type": "plain_text", "text": "GitHub Action job", "emoji": True},
                "url": job_link,
            }

        return [
            {"type": "header", "text": {"type": "plain_text", "text": title, "emoji": True}},
            content,
            {"type": "section", "text": {"type": "mrkdwn", "text": failure_text}},
        ]

    def post_reply(self):
        if self.thread_ts is None:
            raise ValueError("Can only post reply if a post has been made.")

        sorted_dict = sorted(self.doc_test_results.items(), key=lambda t: t[0])
        for job_name, job_result in sorted_dict:
            if len(job_result["failures"]) > 0:
                text = f"*Num failures* :{len(job_result['failed'])} \n"
                failures = job_result["failures"]
                blocks = self.get_reply_blocks(job_name, job_result["job_link"], failures, text=text)

                print("Sending the following reply")
                print(json.dumps({"blocks": blocks}))

                client.chat_postMessage(
                    channel=SLACK_REPORT_CHANNEL_ID,
                    text=f"Results for {job_name}",
                    blocks=blocks,
                    thread_ts=self.thread_ts["ts"],
                )



SUMMARY:
**Class Summary:**

The `Message` class is designed to represent test results and report them in a Slack channel. It has the following primary purposes:

*   Calculate and store metadata about the test results (e.g., number of successes/failures, time spent on tests)
*   Generate reports with various information types (header, failures by category, payload)

**Notable Properties and Methods:**

*   `n_failures`, `n_tests`: Number of test failures and total tests
*   `payload`: A Slack-compatible report payload in JSON format

Key methods:

*   `post()`: Posts the test result report to the specified Slack channel.
*   `error_out()`: Sends a notification with an error message to the same Slack channel.
*   `get_reply_blocks()`: Generates reply blocks for individual test failures, including links to GitHub Action jobs.


key: chroma:document
string_value:
CODE:

def retrieve_artifact(name: str):
    _artifact = {}

    if os.path.exists(name):
        files = os.listdir(name)
        for file in files:
            try:
                with open(os.path.join(name, file), encoding="utf-8") as f:
                    _artifact[file.split(".")[0]] = f.read()
            except UnicodeDecodeError as e:
                raise ValueError(f"Could not open {os.path.join(name, file)}.") from e



SUMMARY:
**Summary**

`retrieve_artifact` function retrieves files from a specified directory and returns an artifact dictionary containing file names as keys and their corresponding contents.

**Main Purpose and Functionality**
The primary purpose of this function is to read the contents files from a given directory named `name`. The function iterates over each file, attempts to decode its contents, and stores them in a dictionary.

```python
def retrieve_artifact(name: str) -> dict:
    # ... (function implementation remains the same)
```

**Key Implementation Details**

* Uses `os.path.exists` to check if the specified directory path exists.
* Lists files using `os.listdir`.
* Tries to open and read each file with UTF-8 encoding, raising an error for decoding failures.

```python
files = os.listdir(name)
for file in files:
    try:
        with open(os.path.join(name, file), encoding="utf-8") as f:
            _artifact[file.split(".")[0]] = f.read()
```

**Notable Dependencies and Requirements**

* `os` module for path manipulation.
* External dependency: `file extensions (e.g., `.txt`, `.pdf`).
* No apparent inheritance or interface implementation.

```python
import os
```

**RAG System Representation**

| Key | Value |
| --- | --- |
| Purpose | Retrieve files from a specified directory and return an artifact dictionary. |
| Functionality | Iterates over each file, attempts to decode its contents, and stores them in a dictionary. |
| Implementation Details | Uses `os.path.exists`, `os.listdir`, and UTF-8 encoding for file reading. |
| Dependencies | `os` module. |


key: chroma:document
string_value:
CODE:

def retrieve_available_artifacts():
    class Artifact:
        def __init__(self, name: str):
            self.name = name
            self.paths = []

        def __str__(self):
            return self.name

        def add_path(self, path: str):
            self.paths.append({"name": self.name, "path": path})

    _available_artifacts: Dict[str, Artifact] = {}

    directories = filter(os.path.isdir, os.listdir())
    for directory in directories:
        artifact_name = directory
        if artifact_name not in _available_artifacts:
            _available_artifacts[artifact_name] = Artifact(artifact_name)

            _available_artifacts[artifact_name].add_path(directory)



SUMMARY:
**Concise Summary:**

This function, `retrieve_available_artifacts`, creates and populates a dictionary of artifacts stored in directories on the system. It utilizes classes to encapsulate artifact details and behavior.

**Key Implementation Details:**

* Creates an `Artifact` class with `name` attribute, which stores directory names as names and their corresponding paths.
* Utilizes `filter()` and `os.path.isdir()` to retrieve only directories from the system.
* Popsulates `_available_artifacts` dictionary with the extracted directory information.

**Notable Dependencies:**

* `os`: for interacting with the operating system (e.g., `os.path.isdir()`)
* `filter()`: a built-in Python function for filtering elements from iterable collections based on condition.

**Primary Purpose and Functionality:**
The main purpose of this code is to retrieve available artifacts stored in directories, which can be utilized for subsequent operations or processing. 

**Additional Informed Insights:**
- The code iterates over all files and folders in the system using `os.listdir()`.
- It populates an internal data structure `_available_artifacts` dictionary, containing only directories as keys.
- It assumes that the directory names match the artifact names within these instances, allowing for consistent categorization.


key: chroma:document
string_value:
CODE:

def clean_model_doc_toc(model_doc: List[dict]) -> List[dict]:
    """
    Cleans a section of the table of content of the model documentation (one specific modality) by removing duplicates
    and sorting models alphabetically.

    Args:
        model_doc (`List[dict]`):
            The list of dictionaries extracted from the `_toctree.yml` file for this specific modality.

    Returns:
        `List[dict]`: List of dictionaries like the input, but cleaned up and sorted.
    """
    counts = defaultdict(int)
    for doc in model_doc:
        counts[doc["local"]] += 1
    duplicates = [key for key, value in counts.items() if value > 1]

    new_doc = []
    for duplicate_key in duplicates:
        titles = list({doc["title"] for doc in model_doc if doc["local"] == duplicate_key})
        if len(titles) > 1:
            raise ValueError(
                f"{duplicate_key} is present several times in the documentation table of content at "
                "`docs/source/en/_toctree.yml` with different *Title* values. Choose one of those and remove the "
                "others."
            )
        # Only add this once
        new_doc.append({"local": duplicate_key, "title": titles[0]})

    # Add none duplicate-keys
    new_doc.extend([doc for doc in model_doc if counts[doc["local"]] == 1])

    # Sort


SUMMARY:
**Concise Summary**

The `clean_model_doc_toc` function cleans the table of content in model documentation by removing duplicates and sorting models alphabetically.

**Main Purpose and Functionality**

* Cleans a list of dictionaries (`model_doc`) extracted from `_toctree.yml` by:
	+ Counting occurrences of each local key
	+ Identifying duplicate keys with multiple titles
	+ Removing all but one version of each duplicate title
* Returns the cleaned list sorted alphabetically by title.

**Key Implementation Details**

* Uses `defaultdict` to count occurrences of each local key.
* Utilizes a lambda function for sorting by title (case-insensitive).
* Raises a ValueError if multiple versions of a title are found for a duplicate key.

**Notable Dependencies or Requirements**

* Uses Python's built-in `List`, `dict`, and `defaultdict` types.
* Assumes the presence of an `_toctree.yml` file with documentation data.


key: chroma:document
string_value:
CODE:

def check_model_doc(overwrite: bool = False):
    """
    Check that the content of the table of content in `_toctree.yml` is clean (no duplicates and sorted for the model
    API doc) and potentially auto-cleans it.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether to just check if the TOC is clean or to auto-clean it (when `overwrite=True`).
    """
    with open(PATH_TO_TOC, encoding="utf-8") as f:
        content = yaml.safe_load(f.read())

    # Get to the API doc
    api_idx = 0
    while content[api_idx]["title"] != "API":
        api_idx += 1
    api_doc = content[api_idx]["sections"]

    # Then to the model doc
    model_idx = 0
    while api_doc[model_idx]["title"] != "Models":
        model_idx += 1

    model_doc = api_doc[model_idx]["sections"]

    # Extract the modalities and clean them one by one.
    modalities_docs = [(idx, section) for idx, section in enumerate(model_doc) if "sections" in section]
    diff = False
    for idx, modality_doc in modalities_docs:
        old_modality_doc = modality_doc["sections"]
        new_modality_doc = clean_model_doc_toc(old_modality_doc)

        if old_modality_doc != new_modality_doc:
            diff = True
            if overwrite:
                model_doc[idx]["sections"] = new_modality_doc

    if diff:
        if overwrite:
            api_doc[model_idx]["sections"] = model_doc
            content[api_idx]["sections"] = api_doc
            with open(PATH_TO_TOC, "w", encoding="utf-8") as f:
                f.write(yaml.dump(content, allow_unicode=True))
        else:
            raise ValueError(
                "The model doc part of the table of content is not properly sorted, run `make style` to fix this."


SUMMARY:
**Summary: check_model_doc function**

The `check_model_doc` function validates and potentially auto-cleans the table of contents (`_toctree.yml`) for the model API doc. It checks for:

* Duplicates: A duplicate title is found, indicating an invalid TOC structure.
* Sorting: The TOC contains unsorted sections.

**Key Implementation Details**

1. Reads `PATH_TO_TOC` file using `yaml.safe_load()`.
2. Iterates to find the "API" and "Models" section indices in the content tree.
3. Extracts modalities documents from the "Models" section.
4. Cleans each modality document using `clean_model_doc_toc()`.

**Notable Dependencies/Requirements**

* `yaml`: for parsing the `_toctree.yml` file.
* `PATH_TO_TOC`: a constant representing the file path to `._toctree.yml`.

**Interfaces and Method Signatures**

1. `yaml.safe_load()`: loads YAML data from a string.
2. `clean_model_doc_toc()`: cleans a model doc using unknown implementation details.

This function appears to be part of a larger project for generating and maintaining documentation, possibly in a MarkDown or Sphinx-based environment.


key: chroma:document
string_value:
CODE:

def create_script(target_test):
    """Create a python script to be run by `git bisect run` to determine if `target_test` passes or fails.
    If a test is not found in a commit, the script with exit code `0` (i.e. `Success`).

    Args:
        target_test (`str`): The test to check.

    Returns:
        `str`: The script to be run by `git bisect run`.
    """

    script = f"""
import os
import subprocess

result = subprocess.run(
    ["python3", "-m", "pytest", "-v", f"{target_test}"],
    capture_output = True,
    text=True,
)
print(result.stdout)

if len(result.stderr) > 0:
    if "ERROR: file or directory not found: " in result.stderr:
        print("test not found in this commit")
        exit(0)
    else:
        print(f"pytest failed to run: {{result.stderr}}")
        exit(-1)
elif f"{target_test} FAILED" in result.stdout:
    print("test failed")
    exit(2)

exit(0)
"""

    with open("target_script.py", "w") as fp:


SUMMARY:
**Summary:**

The `create_script` function generates Python script to be run by `git bisect run` for verifying the outcome of test `target_test`. The script imports required modules, runs the specified test using `pytest`, and outputs the result.

**Key Implementation Details:**

* Runs `python3 -m pytest -v <test_name>` with specified `target_test`
* Checks if error or failure message is present in standard output/error
* Exits with success (0) or failure (-1/2) depending on validation

**Notable Dependencies/Requirements:**

* `os` and `subprocess` modules for executing external commands
* `pytest` module for running tests


key: chroma:document
string_value:
CODE:

def find_bad_commit(target_test, start_commit, end_commit):
    """Find (backward) the earliest commit between `start_commit` and `end_commit` at which `target_test` fails.

    Args:
        target_test (`str`): The test to check.
        start_commit (`str`): The latest commit.
        end_commit (`str`): The earliest commit.

    Returns:
        `str`: The earliest commit at which `target_test` fails.
    """

    if start_commit == end_commit:
        return start_commit

    create_script(target_test=target_test)

    bash = f"""
git bisect reset
git bisect start {start_commit} {end_commit}
git bisect run python3 target_script.py
"""

    with open("run_git_bisect.sh", "w") as fp:
        fp.write(bash.strip())

    result = subprocess.run(
        ["bash", "run_git_bisect.sh"],
        capture_output=True,
        text=True,
    )
    print(result.stdout)

    if "error: bisect run failed" in result.stderr:
        index = result.stderr.find("error: bisect run failed")
        bash_error = result.stderr[index:]

        error_msg = f"Error when running git bisect:\nbash error: {bash_error}"

        pattern = "pytest failed to run: .+"
        pytest_errors = re.findall(pattern, result.stdout)
        if len(pytest_errors) > 0:
            pytest_error = pytest_errors[0]
            index = pytest_error.find("pytest failed to run: ")
            index += len("pytest failed to run: ")
            pytest_error = pytest_error[index:]
            error_msg += f"pytest error: {pytest_error}"

        raise ValueError(error_msg)

    pattern = r"(.+) is the first bad commit"
    commits = re.findall(pattern, result.stdout)

    bad_commit = None
    if len(commits) > 0:
        bad_commit = commits[0]

    print(f"Between `start_commit` {start_commit} and `end_commit` {end_commit}")
    print(f"bad_commit: {bad_commit}\n")



SUMMARY:
**Function Summary: find_bad_commit**

**Purpose:** Find the earliest commit between two given commits at which a specified test fails.

### Functionality:

1.  **Initial Check**: Verifies if `start_commit` and `end_commit` are equal, returning `start_commit` in this case.
2.  Creates a bash script to reset git bisect mode and start it with the provided `target_test`.
3.  Runs the `run_git_bisect.sh` script using `subprocess.run`, capturing its output and error messages.
4.  Checks for errors, specifically "bisect run failed". If found, extracts the error message and continues.
5.  Seeks the first occurrence of "first bad commit" in the standard output to identify the earliest failing commit.

### Key Implementation Details:

*   `git bisect` mode is managed using bash scripts and subprocesses.
*   Standard output and error messages are analyzed for errors or test failures.
*   Regular expressions (`re`) are used to extract specific information from the output.

**Key Method Signatures:**

*   `create_script(target_test)`
*   `subprocess.run(["bash", "run_git_bisect.sh"])`

**Dependencies and Interfaces:**

*   External dependencies:
    *   `git` command
    *   `python3` for running script
    *   `re` (regular expression library) for pattern matching

**Interfaces:**

| Interface | Description                |
| :-------- | -------------------------- |
| `target_test` | Target test to check          |

### Notable Variability:

*   Initial commit check logic.
*   Bash script and subprocess execution.

This analysis highlights the primary purpose and key implementation details of the provided function, facilitating a deeper understanding of its functionality and potential areas for optimization.


key: chroma:document
string_value:
CODE:

def get_commit_info(commit):
    """Get information for a commit via `api.github.com`."""
    pr_number = None
    author = None
    merged_author = None

    url = f"https://api.github.com/repos/huggingface/transformers/commits/{commit}/pulls"
    pr_info_for_commit = requests.get(url).json()

    if len(pr_info_for_commit) > 0:
        pr_number = pr_info_for_commit[0]["number"]

        url = f"https://api.github.com/repos/huggingface/transformers/pulls/{pr_number}"
        pr_for_commit = requests.get(url).json()
        author = pr_for_commit["user"]["login"]
        merged_author = pr_for_commit["merged_by"]["login"]

    if author is None:
        url = f"https://api.github.com/repos/huggingface/transformers/commits/{commit}"
        commit_info = requests.get(url).json()
        author = commit_info["author"]["login"]



SUMMARY:
**Summary**

The `get_commit_info` function retrieves information about a GitHub commit, including the pull request number, author, and merged author. It makes API requests to `api.github.com` to fetch data.

* Purpose: Fetch commit metadata via GitHub API.
* Implementation Details:
	+ Retrieves PR info for a given commit and extracts the PR number.
	+ Fetches details about the associated PR, including author and merged author, if available.
	+ Fallbacks to fetching original commit information using `commit` URL if no PR author is found.
	+ Returns a dictionary with commit-related data.

**Key Method Signsatures/Interfaces**

* `requests.get()`: Used for API requests (`url = f..."`)

**Notable Dependencies**

* `requests`
* `json` ( implicit, handling JSON responses from GitHub API)
* `user`, `login` (accessed through PR and commit responses)

**Interface Inheritance/Implementation**

None.

This function leverages the GitHub API to collect metadata. The implementation caters to both cases when a PR is associated with the commit and when it's not.


key: chroma:document
string_value:
CODE:

def onnx_compliancy(saved_model_path, strict, opset):
    saved_model = SavedModel()
    onnx_ops = []

    with open(os.path.join(REPO_PATH, "utils", "tf_ops", "onnx.json")) as f:
        onnx_opsets = json.load(f)["opsets"]

    for i in range(1, opset + 1):
        onnx_ops.extend(onnx_opsets[str(i)])

    with open(saved_model_path, "rb") as f:
        saved_model.ParseFromString(f.read())

    model_op_names = set()

    # Iterate over every metagraph in case there is more than one (a saved model can contain multiple graphs)
    for meta_graph in saved_model.meta_graphs:
        # Add operations in the graph definition
        model_op_names.update(node.op for node in meta_graph.graph_def.node)

        # Go through the functions in the graph definition
        for func in meta_graph.graph_def.library.function:
            # Add operations in each function
            model_op_names.update(node.op for node in func.node_def)

    # Convert to list, sorted if you want
    model_op_names = sorted(model_op_names)
    incompatible_ops = []

    for op in model_op_names:
        if op not in onnx_ops and op not in INTERNAL_OPS:
            incompatible_ops.append(op)

    if strict and len(incompatible_ops) > 0:
        raise Exception(f"Found the following incompatible ops for the opset {opset}:\n" + incompatible_ops)
    elif len(incompatible_ops) > 0:
        print(f"Found the following incompatible ops for the opset {opset}:")
        print(*incompatible_ops, sep="\n")
    else:


SUMMARY:
**Function Summary: `onnx_compliancy`**

### Purpose and Functionality

The `onnx_compliancy` function checks if a saved model (in ONNX format) is compliant with the specified ONNX opset version. It parses the saved model, extracts its operations, and verifies that they are compatible with the given opset.

**Key Implementation Details**

* Extracts ONNX opsets from `onnx.json` file
* Parses saved model from file using `SavedModel.ParseFromString`
* Iterates over multiple graphs in saved model (if present)
* Compares extracted operation names from saved model to list of compatible operations (`onnx_ops`) and internal operations (`INTERNAL_OPS`)
* Raises exception if strict mode is enabled and incompatible ops are found

**Notable Dependencies and Requirements**

* `SavedModel` class
* `json` module for parsing `onnx.json` file
* `os` module for getting repository path (`REPO_PATH`)
* External dependencies:
	+ `onnx_ops` (list of compatible ONNX opsets)
	+ `INTERNAL_OPS` (list of internal operations)


key: chroma:document
string_value:
CODE:

class QuantizationMessage(Message):
    def __init__(
        self,
        title: str,
        results: Dict,
    ):
        self.title = title

        # Failures and success of the modeling tests
        self.n_success = sum(r["success"] for r in results.values())
        self.single_gpu_failures = sum(r["failed"]["single"] for r in results.values())
        self.multi_gpu_failures = sum(r["failed"]["multi"] for r in results.values())
        self.n_failures = self.single_gpu_failures + self.multi_gpu_failures

        self.n_tests = self.n_failures + self.n_success
        self.results = results
        self.thread_ts = None

    @property
    def payload(self) -> str:
        blocks = [self.header]

        if self.n_failures > 0:
            blocks.append(self.failures_overwiew)
            blocks.append(self.failures_detailed)

        if self.n_failures == 0:
            blocks.append(self.no_failures)

        return json.dumps(blocks)

    @property
    def time(self) -> str:
        all_results = self.results.values()
        time_spent = []
        for r in all_results:
            if len(r["time_spent"]):
                time_spent.extend([x for x in r["time_spent"].split(", ") if len(x.strip())])
        total_secs = 0

        for time in time_spent:
            time_parts = time.split(":")

            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.
            if len(time_parts) == 1:
                time_parts = [0, 0, time_parts[0]]

            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])
            total_secs += hours * 3600 + minutes * 60 + seconds

        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60
        return f"{int(hours)}h{int(minutes)}m{int(seconds)}s"

    @property
    def failures_overwiew(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": (
                    f"There were {self.n_failures} failures, out of {self.n_tests} tests.\n"
                    f"The suite ran in {self.time}."
                ),
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def failures_detailed(self) -> Dict:
        failures = {k: v["failed"] for k, v in self.results.items()}

        individual_reports = []
        for key, value in failures.items():
            device_report = self.get_device_report(value)
            if sum(value.values()):
                report = f"{device_report}{key}"
                individual_reports.append(report)

        header = "Single |  Multi | Category\n"
        failures_report = prepare_reports(
            title="The following quantization tests had failures", header=header, reports=individual_reports
        )

        return {"type": "section", "text": {"type": "mrkdwn", "text": failures_report}}

    def post(self):
        payload = self.payload
        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(payload)}))

        text = f"{self.n_failures} failures out of {self.n_tests} tests," if self.n_failures else "All tests passed."

        self.thread_ts = client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            blocks=payload,
            text=text,
        )

    def post_reply(self):
        if self.thread_ts is None:
            raise ValueError("Can only post reply if a post has been made.")

        for job, job_result in self.results.items():
            if len(job_result["failures"]):
                for device, failures in job_result["failures"].items():
                    blocks = self.get_reply_blocks(
                        job,
                        job_result,
                        failures,
                        device,
                        text=f'Number of failures: {job_result["failed"][device]}',
                    )

                    print("Sending the following reply")
                    print(json.dumps({"blocks": blocks}))

                    client.chat_postMessage(
                        channel="#transformers-ci-daily-quantization",
                        text=f"Results for {job}",
                        blocks=blocks,
                        thread_ts=self.thread_ts["ts"],
                    )


SUMMARY:
**Summary: QuantizationMessage Class**

This class, `QuantizationMessage`, inherits from the `Message` class and represents a message containing quantization testing results.

**Main Purpose and Functionality**
The main purpose of this class is to store and format the results of quantization tests in a JSON payload that can be sent through Slack.

**Implementation Details**

*   **Initialization**: The constructor takes two parameters: `title` (a string) and `results` (a dictionary).
    *   `results`: A dictionary containing test results, including failure information.
        *   `n_success`
        *   `single_gpu_failures`
        *   `multi_gpu_failures`
        *   `n_failures`
        *   `n_tests`
        *   `thread_ts` (initially None)
*   **Properties**:
    *   `payload`: Returns the formatted JSON payload as a string, including optional sections for failure overview and detailed reports.
        *   `blocks`: A list of JSON blocks containing the message content.
        *   Includes failuer section
    *   `time`: Calculates and returns the total time spent on all tests in hours, minutes, and seconds.
*   **Methods**:
    *   `post()`: Sends the message payload to a Slack channel using the client's chat post message method.
    *   `post_reply()`: Posts replies for individual jobs with failures.


key: chroma:document
string_value:
CODE:

def get_new_python_files_between_commits(base_commit: str, commits: List[str]) -> List[str]:
    """
    Get the list of added python files between a base commit and one or several commits.

    Args:
        repo (`git.Repo`):
            A git repository (for instance the Transformers repo).
        base_commit (`str`):
            The commit reference of where to compare for the diff. This is the current commit, not the branching point!
        commits (`List[str]`):
            The list of commits with which to compare the repo at `base_commit` (so the branching point).

    Returns:
        `List[str]`: The list of python files added between a base commit and one or several commits.
    """
    code_diff = []
    for commit in commits:
        for diff_obj in commit.diff(base_commit):
            # We always add new python files
            if diff_obj.change_type == "A" and diff_obj.b_path.endswith(".py"):
                code_diff.append(diff_obj.b_path)



SUMMARY:
**Function Summary:**

`get_new_python_files_between_commits`

Returns a list of added Python files between a base commit (`base_commit`) and one or several commits (`commits`). Compares each commit's diff against the base commit to identify new Python files.

**Key Implementation Details:**

* Uses `git.Repo.diff()` to compare each commit's diff against the base commit.
* Iterates through a list of commits (`commits`) using the `diff()` method.
* Identifies new Python files added in each commit by checking for file creation type ("A") and file extension (`.py`).
* Returns a list of paths to newly created Python files.

**Notable Dependencies/Requirements:**

* `git.Repo`: Git repository access
* `List[str]`: List of commit references

**Primary Purpose:**
Identify new Python files added between a base commit and one or several commits in a Git repository.


key: chroma:document
string_value:
CODE:

def get_new_python_files() -> List[str]:
    """
    Return a list of python files that have been added between the current head and the main branch.

    Returns:
        `List[str]`: The list of python files added.
    """
    repo = Repo(PATH_TO_REPO)

    try:
        # For the cases where the main branch exists locally
        main = repo.refs.main
    except AttributeError:
        # On GitHub Actions runners, it doesn't have local main branch
        main = repo.remotes.origin.refs.main

    print(f"main is at {main.commit}")
    print(f"Current head is at {repo.head.commit}")

    branching_commits = repo.merge_base(main, repo.head)
    for commit in branching_commits:
        print(f"Branching commit: {commit}")


SUMMARY:
**Function Summary**

`get_new_python_files()` returns a list of Python files added between the current Git head and the `main` branch.

**Key Implementation Details**

* Uses `Repo` class to interact with the GitHub repository
* Checks if `main` branch exists locally, falling back to remote `origin` in GitHub Actions runners
* Retrieves merge base commits using `repo.merge_base`
* Returns list of new Python files added using `get_new_python_files_between_commits`

**Notable Dependencies and Requirements**

* `Repo` class from `pathlib` library ( implied but not explicitly stated)
* External dependency on GitHub repository at `PATH_TO_REPO`


key: chroma:document
string_value:
CODE:

def get_new_model():
    new_files = get_new_python_files()
    reg = re.compile(r"src/transformers/models/(.*)/modeling_.*\.py")

    new_model = ""
    for x in new_files:
        find_new_model = reg.findall(x)
        if len(find_new_model) > 0:
            new_model = find_new_model[0]
            # It's unlikely we have 2 new modeling files in a pull request.
            break


SUMMARY:
**Function Summary: `get_new_model`**

Retrieves the names of newly created Python model files in a Git repository.

**Primary Functionality:**
Parses file contents to extract and return the name of the new model file, assuming only one new modeling file exists in each pull request.

**Key Implementation Details:**

* Uses regular expression `reg` to extract pattern "src/transformers/models/(.*)/modeling_.*\.py"
* Iterates over list of new Python files (`new_files`) and uses `re.findall` to find the matching model file name
* Returns the extracted model file name as a string

**Notable Dependencies or Requirements:**

* `re` (regular expression) module for pattern matching
* `get_new_python_files()` external function or call, returning list of new Python files


key: chroma:document
string_value:
CODE:

def parse_message(message: str) -> str:
    """
    Parses a GitHub pull request's comment to find the models specified in it to run slow CI.

    Args:
        message (`str`): The body of a GitHub pull request's comment.

    Returns:
        `str`: The substring in `message` after `run-slow`, run_slow` or run slow`. If no such prefix is found, the
        empty string is returned.
    """
    if message is None:
        return ""

    message = message.strip().lower()

    # run-slow: model_1, model_2
    if not message.startswith(("run-slow", "run_slow", "run slow")):
        return ""
    message = message[len("run slow") :]
    # remove leading `:`
    while message.strip().startswith(":"):
        message = message.strip()[1:]



SUMMARY:
**Summary**

### Functionality

The `parse_message` function parses a GitHub pull request comment to extract models specified in the message to run slow CI.

### Implementation Details

* **Key Algorithm:** Trims and normalizes user input message.
* **Method Signature:**
	+ `parse_message(message: str) -> str`
* **Design Choice:** Uses multiple prefixes ("run-slow", "run_slow", "run slow") to account for varying notation in comments.

### Dependencies and Requirements

* None explicitly listed, but assumes `str` type for input message.


key: chroma:document
string_value:
CODE:

def get_models(message: str):
    models = parse_message(message)


SUMMARY:
**Summary**

### Function Overview

`get_models(message: str)`

Extracts and preprocesses a list of models from a given message.

### Primary Purpose

Resets and splits the `models` string to separate individual model names, replacing commas with spaces.

### Key Implementation Details

* Uses `parse_message(message)` (exact class/function not provided)
* Applies whitespace manipulation to output string
* Utilizes string splitting (`split()`) on `models`

### Notable Dependencies or Requirements

* `parse_message(message)` (related function not provided)
 
### External Dependencies

List of external dependencies is not specified in the given code chunk


key: chroma:document
string_value:
CODE:

def check_model_names(model_name: str):
    allowed = string.ascii_letters + string.digits + "_"


SUMMARY:
**Summary**

Checks if a `model_name` is valid. A valid model name consists of letters, digits, and underscores only.

* **Primary Purpose**: Validate model names to ensure they comply with the specified format.
* **Key Implementation Details**:
	+ Uses regular expression for string validation (`string.ascii_letters + string.digits + "_"`).
	+ Checks for leading/trailing underscore prefixes using `startswith()` and `endswith()`.
* **Notable Dependencies**:
	+ `string` module for character classification.
* **Interfaces/Inherited Class**: None.

In RAG system notation: 
### model_name_validation
- Description: Validates a model name to ensure it contains only letters, digits, or underscores.
- Primary Purpose: Ensure model names meet the specified format requirement.


key: chroma:document
string_value:
CODE:

def process_file(modular_file_path, generated_modeling_content, file_type="modeling_", fix_and_overwrite=False):
    file_name_prefix = file_type.split("*")[0]
    file_name_suffix = file_type.split("*")[-1] if "*" in file_type else ""
    file_path = modular_file_path.replace("modular_", f"{file_name_prefix}_").replace(".py", f"{file_name_suffix}.py")
    # Read the actual modeling file
    with open(file_path, "r") as modeling_file:
        content = modeling_file.read()
    output_buffer = StringIO(generated_modeling_content[file_type][0])
    output_buffer.seek(0)
    output_content = output_buffer.read()
    diff = difflib.unified_diff(
        output_content.splitlines(),
        content.splitlines(),
        fromfile=f"{file_path}_generated",
        tofile=f"{file_path}",
        lineterm="",
    )
    diff_list = list(diff)
    # Check for differences
    if diff_list:
        if fix_and_overwrite:
            with open(file_path, "w") as modeling_file:
                modeling_file.write(generated_modeling_content[file_type][0])
            console.print(f"[bold blue]Overwritten {file_path} with the generated content.[/bold blue]")
        else:
            console.print(f"\n[bold red]Differences found between the generated code and {file_path}:[/bold red]\n")
            diff_text = "\n".join(diff_list)
            syntax = Syntax(diff_text, "diff", theme="ansi_dark", line_numbers=True)
            console.print(syntax)
        return 1
    else:
        console.print(f"[bold green]No differences found for {file_path}.[/bold green]")


SUMMARY:
**Function Summary**

`process_file(modular_file_path, generated_modeling_content, file_type="modeling_", fix_and_overwrite=False)`

**Purpose and Functionality**
Process and compare the content of two modeling files located in `modular_file_path`.

**Main Algorithms/P Patterns/Design Choices**

* Reading and comparing file contents using `difflib.unified_diff` to identify differences.
* Overwriting or printing differences based on `fix_and_overwrite` flag.

**Key Method Signatures and Interfaces**

- `difflib.unified_diff(fromfile, tofile, lineterm="")`
- `console.print()`

**Notable Dependencies and Requirements**

- `StringIO`
- `difflib`
- `console` (assuming a custom console object or module)
- `generated_modeling_content[file_type]` referencing an external data structure.


key: chroma:document
string_value:
CODE:

def compare_files(modular_file_path, fix_and_overwrite=False):
    # Generate the expected modeling content
    generated_modeling_content = convert_modular_file(modular_file_path)
    diff = 0
    for file_type in generated_modeling_content.keys():
        diff += process_file(modular_file_path, generated_modeling_content, file_type, fix_and_overwrite)


SUMMARY:
**Summary:**

`compare_files` function compares the expected modeling content with actual files specified by `modular_file_path`. It iterates through each file type, processing them using `process_file`, and returns the cumulative difference (`diff`). 

**Key Implementation Details:**

* Generates expected modeling content using `convert_modular_file(modular_file_path)`
	+ Input parameter: `modular_file_path`
	+ Output parameter: `generated_modeling_content` dictionary containing different file types
* Iterates through each file type in `generated_modeling_content` and processes them using `process_file(modular_file_path, generated_modeling_content, file_type, fix_and_overwrite)`
	+ Input parameters: `modular_file_path`, `generated_modeling_content`, `file_type`
	+ Output parameter: cumulative difference (`diff`)
* Returns the cumulative difference (`diff`) after processing all files

**Notable Dependencies/Requirements:**

* `convert_modular_file` function
* `process_file` function with parameters `modular_file_path`, `generated_modeling_content`, `file_type`, and `fix_and_overwrite`
 
This summary aims to preserve the original method's flow, focus on key implementation details, and reference related functions by exact name.


key: chroma:document
string_value:
CODE:

def find_backend(line: str) -> Optional[str]:
    """
    Find one (or multiple) backend in a code line of the init.

    Args:
        line (`str`): A code line of the main init.

    Returns:
        Optional[`str`]: If one (or several) backend is found, returns it. In the case of multiple backends (the line
        contains `if is_xxx_available() and `is_yyy_available()`) returns all backends joined on `_and_` (so
        `xxx_and_yyy` for instance).
    """
    if _re_test_backend.search(line) is None:
        return None
    backends = [b[0] for b in _re_backend.findall(line)]
    backends.sort()


SUMMARY:
**Summary**

`find_backend` function identifies and extracts multiple backend names (`xxx_and_yyy`) from a given code line of the main `init` file. It performs a case-insensitive search using regular expression `_re_test_backend.search(line)`.

**Implementation Details**

* Uses two regular expressions: `_re_test_backend` to search for the backend pattern and `_re_backend` to extract the backend names.
* Extracts backend names from the line using a list comprehension (`[b[0] for b in _re_backend.findall(line)]`) and sorts them alphabetically.

**Return Value**

Returns all extracted backend names joined on `_and_`, or `None` if no backend is found (`if _re_test_backend.search(line) is None`).


key: chroma:document
string_value:
CODE:

def parse_init(init_file) -> Optional[Tuple[Dict[str, List[str]], Dict[str, List[str]]]]:
    """
    Read an init_file and parse (per backend) the `_import_structure` objects defined and the `TYPE_CHECKING` objects
    defined.

    Args:
        init_file (`str`): Path to the init file to inspect.

    Returns:
        `Optional[Tuple[Dict[str, List[str]], Dict[str, List[str]]]]`: A tuple of two dictionaries mapping backends to list of
        imported objects, one for the `_import_structure` part of the init and one for the `TYPE_CHECKING` part of the
        init. Returns `None` if the init is not a custom init.
    """
    with open(init_file, "r", encoding="utf-8", newline="\n") as f:
        lines = f.readlines()

    # Get the to `_import_structure` definition.
    line_index = 0
    while line_index < len(lines) and not lines[line_index].startswith("_import_structure = {"):
        line_index += 1

    # If this is a traditional init, just return.
    if line_index >= len(lines):
        return None

    # First grab the objects without a specific backend in _import_structure
    objects = []
    while not lines[line_index].startswith("if TYPE_CHECKING") and find_backend(lines[line_index]) is None:
        line = lines[line_index]
        # If we have everything on a single line, let's deal with it.
        if _re_one_line_import_struct.search(line):
            content = _re_one_line_import_struct.search(line).groups()[0]
            imports = re.findall(r"\[([^\]]+)\]", content)
            for imp in imports:
                objects.extend([obj[1:-1] for obj in imp.split(", ")])
            line_index += 1
            continue
        single_line_import_search = _re_import_struct_key_value.search(line)
        if single_line_import_search is not None:
            imports = [obj[1:-1] for obj in single_line_import_search.groups()[0].split(", ") if len(obj) > 0]
            objects.extend(imports)
        elif line.startswith(" " * 8 + '"'):
            objects.append(line[9:-3])
        line_index += 1

    # Those are stored with the key "none".
    import_dict_objects = {"none": objects}

    # Let's continue with backend-specific objects in _import_structure
    while not lines[line_index].startswith("if TYPE_CHECKING"):
        # If the line is an if not is_backend_available, we grab all objects associated.
        backend = find_backend(lines[line_index])
        # Check if the backend declaration is inside a try block:
        if _re_try.search(lines[line_index - 1]) is None:
            backend = None

        if backend is not None:
            line_index += 1

            # Scroll until we hit the else block of try-except-else
            while _re_else.search(lines[line_index]) is None:
                line_index += 1

            line_index += 1

            objects = []
            # Until we unindent, add backend objects to the list
            while len(lines[line_index]) <= 1 or lines[line_index].startswith(" " * 4):
                line = lines[line_index]
                if _re_import_struct_add_one.search(line) is not None:
                    objects.append(_re_import_struct_add_one.search(line).groups()[0])
                elif _re_import_struct_add_many.search(line) is not None:
                    imports = _re_import_struct_add_many.search(line).groups()[0].split(", ")
                    imports = [obj[1:-1] for obj in imports if len(obj) > 0]
                    objects.extend(imports)
                elif _re_between_brackets.search(line) is not None:
                    imports = _re_between_brackets.search(line).groups()[0].split(", ")
                    imports = [obj[1:-1] for obj in imports if len(obj) > 0]
                    objects.extend(imports)
                elif _re_quote_object.search(line) is not None:
                    objects.append(_re_quote_object.search(line).groups()[0])
                elif line.startswith(" " * 8 + '"'):
                    objects.append(line[9:-3])
                elif line.startswith(" " * 12 + '"'):
                    objects.append(line[13:-3])
                line_index += 1

            import_dict_objects[backend] = objects
        else:
            line_index += 1

    # At this stage we are in the TYPE_CHECKING part, first grab the objects without a specific backend
    objects = []
    while (
        line_index < len(lines)
        and find_backend(lines[line_index]) is None
        and not lines[line_index].startswith("else")
    ):
        line = lines[line_index]
        single_line_import_search = _re_import.search(line)
        if single_line_import_search is not None:
            objects.extend(single_line_import_search.groups()[0].split(", "))
        elif line.startswith(" " * 8):
            objects.append(line[8:-2])
        line_index += 1

    type_hint_objects = {"none": objects}

    # Let's continue with backend-specific objects
    while line_index < len(lines):
        # If the line is an if is_backend_available, we grab all objects associated.
        backend = find_backend(lines[line_index])
        # Check if the backend declaration is inside a try block:
        if _re_try.search(lines[line_index - 1]) is None:
            backend = None

        if backend is not None:
            line_index += 1

            # Scroll until we hit the else block of try-except-else
            while _re_else.search(lines[line_index]) is None:
                line_index += 1

            line_index += 1

            objects = []
            # Until we unindent, add backend objects to the list
            while len(lines[line_index]) <= 1 or lines[line_index].startswith(" " * 8):
                line = lines[line_index]
                single_line_import_search = _re_import.search(line)
                if single_line_import_search is not None:
                    objects.extend(single_line_import_search.groups()[0].split(", "))
                elif line.startswith(" " * 12):
                    objects.append(line[12:-2])
                line_index += 1

            type_hint_objects[backend] = objects
        else:
            line_index += 1



SUMMARY:
**Function Summary: `parse_init`**

Parsing an init file to extract import and TYPE_CHECKING definitions.

**Purpose:** Read an init file, identifying and parsing the `_import_structure` and `TYPE_CHECKING` objects defined in it.

**Primary Purpose:** The function reads a `.init` file and returns two dictionaries mapping backends to lists of imported objects: one for the `_import_structure` section and one for the `TYPE_CHECKING` section.

**Key Implementation Details:**

*   Checks for `_import_structure = { ... }` definitions to parse backend-specific imports.
*   Iterates through lines to identify and extract:
    *   Import structures without a specific backend (`_re_one_line_import_struct.search`)
    *   Backend-specific imports within the try-except-else block (`_re_backend_find`)
    *   TYPE_CHECKING objects without a specific backend (`_re_import.search`)
*   Parses import definitions using regular expressions (e.g., `_import_struct_add_many`, `_re_between_brackets`)

**Notable Dependencies and Requirements:**

*   The function uses regular expression classes (`_re_one_line_import_struct`, `_re_backend_find`, etc.) to parse the init file.
*   It relies on `find_backend()` function, but its implementation is not provided in this code snippet.

**Return Value:** A tuple of two dictionaries:
    *   `import_dict_objects`: Mapping backends to lists of import structures without a specific backend (`"none"`).
    *   `type_hint_objects`: Mapping backends (if applicable) to lists of TYPE_CHECKING objects without a specific backend (`"none"`).


key: chroma:document
string_value:
CODE:

def analyze_results(import_dict_objects: Dict[str, List[str]], type_hint_objects: Dict[str, List[str]]) -> List[str]:
    """
    Analyze the differences between _import_structure objects and TYPE_CHECKING objects found in an init.

    Args:
        import_dict_objects (`Dict[str, List[str]]`):
            A dictionary mapping backend names (`"none"` for the objects independent of any specific backend) to
            list of imported objects.
        type_hint_objects (`Dict[str, List[str]]`):
            A dictionary mapping backend names (`"none"` for the objects independent of any specific backend) to
            list of imported objects.

    Returns:
        `List[str]`: The list of errors corresponding to mismatches.
    """

    def find_duplicates(seq):
        return [k for k, v in collections.Counter(seq).items() if v > 1]

    # If one backend is missing from the other part of the init, error early.
    if list(import_dict_objects.keys()) != list(type_hint_objects.keys()):
        return ["Both sides of the init do not have the same backends!"]

    errors = []
    # Find all errors.
    for key in import_dict_objects.keys():
        # Duplicate imports in any half.
        duplicate_imports = find_duplicates(import_dict_objects[key])
        if duplicate_imports:
            errors.append(f"Duplicate _import_structure definitions for: {duplicate_imports}")
        duplicate_type_hints = find_duplicates(type_hint_objects[key])
        if duplicate_type_hints:
            errors.append(f"Duplicate TYPE_CHECKING objects for: {duplicate_type_hints}")

        # Missing imports in either part of the init.
        if sorted(set(import_dict_objects[key])) != sorted(set(type_hint_objects[key])):
            name = "base imports" if key == "none" else f"{key} backend"
            errors.append(f"Differences for {name}:")
            for a in type_hint_objects[key]:
                if a not in import_dict_objects[key]:
                    errors.append(f"  {a} in TYPE_HINT but not in _import_structure.")
            for a in import_dict_objects[key]:
                if a not in type_hint_objects[key]:
                    errors.append(f"  {a} in _import_structure but not in TYPE_HINT.")


SUMMARY:
**Function Summary**

`analyze_results` analyzes the discrepancies between `_import_structure` objects and `TYPE_CHECKING` objects found in an init.

**Main Purpose**
Identifies duplicate imports, missing imports, and incorrect backend names across both structures.

**Key Implementation Details**

* Verifies that both input dictionaries have identical keys (backends).
* Uses a custom function `find_duplicates` to detect duplicate values in lists.
* Iterates through each key-value pair, comparing the two dictionaries for:

    * Duplicate imports
    * Missing imports

**Notable Dependencies/Requirements**

* Python built-in libraries: `collections`
* External dependencies:
	+ None (self-contained)
* Implemented interfaces:
	+ No explicit interface implementation; relies on dictionary comparisons.


key: chroma:document
string_value:
CODE:

def check_all_inits():
    """
    Check all inits in the transformers repo and raise an error if at least one does not define the same objects in
    both halves.
    """
    failures = []
    for root, _, files in os.walk(PATH_TO_TRANSFORMERS):
        if "__init__.py" in files:
            fname = os.path.join(root, "__init__.py")
            objects = parse_init(fname)
            if objects is not None:
                errors = analyze_results(*objects)
                if len(errors) > 0:
                    errors[0] = f"Problem in {fname}, both halves do not define the same objects.\n{errors[0]}"
                    failures.append("\n".join(errors))
    if len(failures) > 0:


SUMMARY:
**Function Summary:**

`check_all_inits()`

Checks all initalization files (`__init__.py`) in the transformers repository for potential object definition discrepancies between both halves. Raises an error if any file fails to define the same objects.

**Key Implementation Details:**

* Iterates through `__init__.py` files in the transformers repository using `os.walk`.
* Parses initialization file contents using `parse_init(fname)`.
* Analyzes results using `analyze_results(*objects)` and accumulates errors.
* Raises a `ValueError` with error messages if any discrepancies are found.

**Notable Dependencies:**

* `os`, `PATH_TO_TRANSFORMERS`
* `parse_init`, `analyze_results` (external functions not shown in this code snippet)


key: chroma:document
string_value:
CODE:

def get_transformers_submodules() -> List[str]:
    """
    Returns the list of Transformers submodules.
    """
    submodules = []
    for path, directories, files in os.walk(PATH_TO_TRANSFORMERS):
        for folder in directories:
            # Ignore private modules
            if folder.startswith("_"):
                directories.remove(folder)
                continue
            # Ignore leftovers from branches (empty folders apart from pycache)
            if len(list((Path(path) / folder).glob("*.py"))) == 0:
                continue
            short_path = str((Path(path) / folder).relative_to(PATH_TO_TRANSFORMERS))
            submodule = short_path.replace(os.path.sep, ".")
            submodules.append(submodule)
        for fname in files:
            if fname == "__init__.py":
                continue
            short_path = str((Path(path) / fname).relative_to(PATH_TO_TRANSFORMERS))
            submodule = short_path.replace(".py", "").replace(os.path.sep, ".")
            if len(submodule.split(".")) == 1:
                submodules.append(submodule)


SUMMARY:
**Summary**

`get_transformers_submodules()` function returns a list of Transformers submodules by traversing the directory tree rooted at `PATH_TO_TRANSFORMERS`. It filters out private modules, empty folders, and initializes that are not submodule names.

**Key Implementation Details**

* Uses `os.walk()` to traverse directory tree
* Checks for file patterns in folders (`*.py`, `__init__.py`) and files (excluding `__init__.py`)
* Normalizes folder and file paths to standardize submodule names

**Notable Dependencies or Requirements**

* `PATH_TO_TRANSFORMERS`
* `os` module for walking directories
* `Path` class from `pathlib` library for path manipulation


key: chroma:document
string_value:
CODE:

def check_submodules():
    """
    Check all submodules of Transformers are properly registered in the main init. Error otherwise.
    """
    # This is to make sure the transformers module imported is the one in the repo.
    from transformers.utils import direct_transformers_import

    transformers = direct_transformers_import(PATH_TO_TRANSFORMERS)

    import_structure_keys = set(transformers._import_structure.keys())
    # This contains all the base keys of the _import_structure object defined in the init, but if the user is missing
    # some optional dependencies, they may not have all of them. Thus we read the init to read all additions and
    # (potentiall re-) add them.
    with open(os.path.join(PATH_TO_TRANSFORMERS, "__init__.py"), "r") as f:
        init_content = f.read()
    import_structure_keys.update(set(re.findall(r"import_structure\[\"([^\"]*)\"\]", init_content)))

    module_not_registered = [
        module
        for module in get_transformers_submodules()
        if module not in IGNORE_SUBMODULES and module not in import_structure_keys
    ]

    if len(module_not_registered) > 0:
        list_of_modules = "\n".join(f"- {module}" for module in module_not_registered)
        raise ValueError(
            "The following submodules are not properly registed in the main init of Transformers:\n"
            f"{list_of_modules}\n"
            "Make sure they appear somewhere in the keys of `_import_structure` with an empty list as value."


SUMMARY:
**Summary: `check_submodules()` Function**

* **Purpose:** Validate that all submodules in `Transformers` are properly initialized with their respective import structures.
* **Key Implementation Details:**
	+ Verifies `transformers._import_structure` keys against the expected structure and user-defined imports from `_init.py`.
	+ Compares submodule lists against imported `import_structure_keys`.
	+ Raises a `ValueError` if any submodules are not properly registered.
* **Notable Dependencies & Requirements:**
	+ `direct_transformers_import`: imports transformers module ensuring it's the one in the repo.
	+ `get_transformers_submodules`: recursively finds all submodules of Transformers.
	+ `_init.py`: provides user-defined import structure additions for optional dependencies.


key: chroma:document
string_value:
CODE:

def get_daily_ci_runs(token, num_runs=7):
    """Get the workflow runs of the scheduled (daily) CI.

    This only selects the runs triggered by the `schedule` event on the `main` branch.
    """
    headers = None
    if token is not None:
        headers = {"Accept": "application/vnd.github+json", "Authorization": f"Bearer {token}"}

    # The id of a workflow (not of a workflow run).
    # From a given workflow run (where we have workflow run id), we can get the workflow id by going to
    # https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}
    # and check the `workflow_id` key.
    workflow_id = "90575235"

    url = f"https://api.github.com/repos/huggingface/transformers/actions/workflows/{workflow_id}/runs"
    # On `main` branch + event being `schedule` + not returning PRs + only `num_runs` results
    url += f"?branch=main&event=schedule&exclude_pull_requests=true&per_page={num_runs}"

    result = requests.get(url, headers=headers).json()



SUMMARY:
**Function Summary: `get_daily_ci_runs`**

**Purpose and Functionality:** Retrieves the workflow run history of scheduled CI for the GitHub repository "huggingface/transformers" on the `main` branch filtered by schedule events.

**Key Implementation Details:**

* Employs the API endpoint `https://api.github.com/repos/huggingface/transformers/actions/workflows/{workflow_id}/runs` with query parameters:
	+ Branch: `main`
	+ Event: `schedule`
	+ Exclude pull requests: `true`
	+ Page size: `num_runs`
* Utilizes the `requests.get()` method to send a GET request to the API endpoint, authenticated via Bearer token.

**Notable Dependencies and Requirements:**

* `token`: GitHub API authentication token
* `requests` library for API calls
* External dependencies:
	+ github.com ( GitHub repository)
	+ transformrsactions API


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_runs(token):
    """Get the last completed workflow run id of the scheduled (daily) CI."""
    workflow_runs = get_daily_ci_runs(token)
    workflow_run_id = None
    for workflow_run in workflow_runs:
        if workflow_run["status"] == "completed":
            workflow_run_id = workflow_run["id"]
            break



SUMMARY:
**Summary:**

```markdown
### Function:
#### `get_last_daily_ci_runs(token)`

#### Purpose:
 Retrieves the ID of the most recent completed daily Continuous Integration (CI) workflow.

#### Implementation Details:

* Calls `get_daily_ci_runs(token)` to retrieve daily CI runs
* Iterates through runs, returning the ID of the first complete run (`if "status" in workflow_run:`)
* Returns the most recent completed daily CI run's ID

#### Notable Dependencies:
None

#### Interfaces:
- `get_daily_ci_runs(token)`
```

**RAG Code Tag:**
`func: get_last_daily_ci_runs(token)`:


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_run_commit(token):
    """Get the commit sha of the last completed scheduled daily CI workflow run."""
    workflow_runs = get_daily_ci_runs(token)
    head_sha = None
    for workflow_run in workflow_runs:
        if workflow_run["status"] == "completed":
            head_sha = workflow_run["head_sha"]
            break



SUMMARY:
**Summary**

The `get_last_daily_ci_run_commit` function retrieves the commit SHA of the last completed scheduled daily CI workflow run.

**Key Points:**

* **Purpose:** Returns the SHA of the latest completed CI run.
* **Functionality:** Iterates through completed daily CI runs to find the head SHA associated with the completion.
* **Implementation Details:**
	+ Calls `get_daily_ci_runs` function to retrieve successful workflow runs.
	+ Loops through runs, identifying completed ones and saving their corresponding head SHA.
	+ Returns the head SHA of the last completed run.

**Notable Dependencies/Requirements:**

* `get_daily_ci_runs(token)` - Function called to retrieve daily CI runs.


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_artifacts(artifact_names, output_dir, token):
    """Get the artifacts of last completed workflow run id of the scheduled (daily) CI."""
    workflow_run_id = get_last_daily_ci_runs(token)
    if workflow_run_id is not None:
        artifacts_links = get_artifacts_links(worflow_run_id=workflow_run_id, token=token)
        for artifact_name in artifact_names:
            if artifact_name in artifacts_links:
                artifact_url = artifacts_links[artifact_name]
                download_artifact(
                    artifact_name=artifact_name, artifact_url=artifact_url, output_dir=output_dir, token=token


SUMMARY:
**Function Summary:**

`get_last_daily_ci_artifacts`

 Retrieves and downloads artifacts from the last completed daily CI workflow run.

**Key Implementation Details:**

* Fetches `workflow_run_id` using `get_last_daily_ci_runs`.
* Retrieves artifact links for the specified `workflow_run_id` using `get_artifacts_links`.
* Iterates over `artifact_names`, downloading artifacts matching those names using `download_artifact`.

**Notable Dependencies or Requirements:**

* Utilizes `token` in multiple calls.
* Calls `get_artifacts_links` with `workflow_run_id` and `token`.
* Expects implementation of `download_artifact` function separate from this snippet.

This summary prescribes the main purpose, key algorithms/patterns, and dependencies used within `get_last_daily_ci_artifacts`.


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_reports(artifact_names, output_dir, token):
    """Get the artifacts' content of the last completed workflow run id of the scheduled (daily) CI."""
    get_last_daily_ci_artifacts(artifact_names, output_dir, token)

    results = {}
    for artifact_name in artifact_names:
        artifact_zip_path = os.path.join(output_dir, f"{artifact_name}.zip")
        if os.path.isfile(artifact_zip_path):
            results[artifact_name] = {}
            with zipfile.ZipFile(artifact_zip_path) as z:
                for filename in z.namelist():
                    if not os.path.isdir(filename):
                        # read the file
                        with z.open(filename) as f:
                            results[artifact_name][filename] = f.read().decode("UTF-8")



SUMMARY:
**Summary**

The `get_last_daily_ci_reports` function retrieves the artifacts' content from the last completed daily CI workflow run. It takes three parameters: `artifact_names`, `output_dir`, and `token`.

**Purpose and Functionality**

Retrieves artifacts from the last daily CI workflow run, saving their contents in a dictionary for each artifact.

**Key Implementation Details**

* Iterates over `.zip` files with artifact names matching in the provided directories.
* Reads and decodes file contents using UTF-8 encoding.
* Returns a nested dictionary containing artifact names as keys and their corresponding file contents as values.

**Notable Dependencies or Requirements**

* `os.path.isfile`
* `os.open`
* `zipped zipfile.ZipFile`

Note: This summary assumes the absence of any external dependencies not mentioned in the provided code. If such dependencies exist, please provide them for a more comprehensive analysis.


key: chroma:document
string_value:
CODE:

def extract_time_from_single_job(job):
    """Extract time info from a single job in a GitHub Actions workflow run"""

    job_info = {}

    start = job["started_at"]
    end = job["completed_at"]

    start_datetime = date_parser.parse(start)
    end_datetime = date_parser.parse(end)

    duration_in_min = round((end_datetime - start_datetime).total_seconds() / 60.0)

    job_info["started_at"] = start
    job_info["completed_at"] = end
    job_info["duration"] = duration_in_min



SUMMARY:
**Summary**

This function, `extract_time_from_single_job`, extracts time-related information from a single job in a GitHub Actions workflow run. It:

* Retrieves the start and end timestamps of the job
* Calculates the duration in minutes between these two timestamps
* Stores this duration along with the original timestamps in a dictionary, `job_info`
* Returns the populated `job_info` dictionary

**Key Implementation Details**

* Retrieves dates using date parsing (`date_parser.parse`)
* Calculates duration in minutes from the timestamp difference ( converted to seconds)

**Notable Dependencies or Requirements**

* `date_parser`: assumed dependency for parsing timestamps into datetimes
* Expected input: `job` dictionary with keys "started_at" and "completed_at"
* Returns a dictionary (`job_info`) with keys "started_at", "completed_at", and "duration".


key: chroma:document
string_value:
CODE:

def get_job_time(workflow_run_id, token=None):
    """Extract time info for all jobs in a GitHub Actions workflow run"""

    headers = None
    if token is not None:
        headers = {"Accept": "application/vnd.github+json", "Authorization": f"Bearer {token}"}

    url = f"https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}/jobs?per_page=100"
    result = requests.get(url, headers=headers).json()
    job_time = {}

    try:
        job_time.update({job["name"]: extract_time_from_single_job(job) for job in result["jobs"]})
        pages_to_iterate_over = math.ceil((result["total_count"] - 100) / 100)

        for i in range(pages_to_iterate_over):
            result = requests.get(url + f"&page={i + 2}", headers=headers).json()
            job_time.update({job["name"]: extract_time_from_single_job(job) for job in result["jobs"]})

        return job_time
    except Exception:
        print(f"Unknown error, could not fetch links:\n{traceback.format_exc()}")



SUMMARY:
**Function Summary**

`get_job_time`: Extracts time information for all jobs in a GitHub Actions workflow run.

**Primary Purpose:** Retrieves and extracts time info from multiple GitHub Actions workflow jobs.

**Implementation Details:**

- Uses `requests` to fetch GitHub API data
- Iterates through 2 pages of jobs (due to pagination) to collect time info
- Utilizes `extract_time_from_single_job` function (not shown in this snippet)
- Handles exceptions and prints error messages if an issue occurs

**Design Choices:**

* Uses Bearer token authentication for API requests
* Sets a page size of 100 jobs per request (per GitHub's rate limiting guidelines)

**Notable Dependencies:**

* `requests`
* `math` (for calculating total pages to iterate over)
* `extract_time_from_single_job` (not shown in this snippet, assumed to be implemented elsewhere)


key: chroma:document
string_value:
CODE:

def get_last_stable_minor_release():
    # Get the last stable release of transformers
    url = "https://pypi.org/pypi/transformers/json"
    release_data = requests.get(url).json()

    # Find the last stable release of of transformers (version below current version)
    major_version, minor_version, patch_version, _ = current_version.split(".")
    last_major_minor = f"{major_version}.{int(minor_version) - 1}"
    last_stable_minor_releases = [
        release for release in release_data["releases"] if release.startswith(last_major_minor)
    ]
    last_stable_release = sorted(last_stable_minor_releases, key=version.parse)[-1]



SUMMARY:
**Summary:**

The `get_last_stable_minor_release` function retrieves the last stable minor release of the `transformers` library by querying the PyPI JSON API. It compares the current major version to determine the last stable major and minor version, then filters releases to find those with versions below this threshold.

**Key Implementation Details:**

* Fetches `transformers` Pypi JSON data using `requests`
* Compares current version to find last stable minor release
* Uses `version.parse()` to sort and compare release versions

**Notable Dependencies/Requirements:**

* `requests` library for HTTP client
* `pyperpapi` library (`version.parse()`) for parsing release versions


key: chroma:document
string_value:
CODE:

def build_tip_message(last_stable_release):
    return (
        """
<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
"""
        + f"""If you run into any issues running this model, please reinstall the last version that supported this model: v{last_stable_release}.
You can do so by running the following command: `pip install -U transformers=={last_stable_release}`.

</Tip>"""


SUMMARY:
**Function Summary: build_tip_message**

* **Purpose**: Generates a maintenance mode warning message for a machine learning model, indicating that new PRs changing the code are not accepted.
	+ Primary purpose: Warn users about model maintenance mode and provide instructions to reinstall previous version.

**Key Implementation Details:**
- Uses f-string formatting to insert `last_stable_release` into the message.
- Includes a Git installation command for reusing the last stable version supported by the model (Transformer vX.X).

**Notable Dependencies or Requirements:** None mentioned explicitly.


key: chroma:document
string_value:
CODE:

def insert_tip_to_model_doc(model_doc_path, tip_message):
    tip_message_lines = tip_message.split("\n")

    with open(model_doc_path, "r") as f:
        model_doc = f.read()

    # Add the tip message to the model doc page directly underneath the title
    lines = model_doc.split("\n")

    new_model_lines = []
    for line in lines:
        if line.startswith("# "):
            new_model_lines.append(line)
            new_model_lines.extend(tip_message_lines)
        else:
            new_model_lines.append(line)

    with open(model_doc_path, "w") as f:


SUMMARY:
**Summary:**

`insert_tip_to_model_doc` function appends a tip message below the title section of an existing Markdown document file.

**Key Details:**

* Reads an input Markdown document file (`model_doc_path`) and tip message (`tip_message`)
* Splits the document into lines, separating those with title markers (`# `) from others
* Inserts the tip message lines below the original title lines, preserving blank line separation

**Implementation Notes:**

* Uses `file read/write` operations to modify an external file resource.

**External Dependencies: None specified. No known inherited or implemented interfaces are assumed to be used in this code snippet.**


key: chroma:document
string_value:
CODE:

def get_model_doc_path(model: str) -> Tuple[Optional[str], Optional[str]]:
    # Possible variants of the model name in the model doc path
    model_names = [model, model.replace("_", "-"), model.replace("_", "")]

    model_doc_paths = [REPO_PATH / f"docs/source/en/model_doc/{model_name}.md" for model_name in model_names]

    for model_doc_path, model_name in zip(model_doc_paths, model_names):
        if os.path.exists(model_doc_path):
            return model_doc_path, model_name



SUMMARY:
**Summary:**

This function retrieves the path to an English model documentation file based on the provided `model` name. It checks for three possible variant names and searches for a non-empty documentation directory within the repository.

**Implementation Details:**

* The function returns two optional strings, (`model_doc_path`, `model_name`) if a matching documentation file exists.
* The search considers three variant names:
	1. Original model name
	2. Model name with hyphens instead of underscores
	3. Model name without hyphens
* It utilizes the `os` module for path existence checks.

**Notable Dependencies:**

* `REPO_PATH`: assumed to be a constants or an external variable referencing the repository's root directory.
* `os.path.exists()`
* `zip()` function


key: chroma:document
string_value:
CODE:

def extract_model_info(model):
    model_info = {}
    model_doc_path, model_doc_name = get_model_doc_path(model)
    model_path = REPO_PATH / f"src/transformers/models/{model}"

    if model_doc_path is None:
        print(f"Model doc path does not exist for {model}")
        return None
    model_info["model_doc_path"] = model_doc_path
    model_info["model_doc_name"] = model_doc_name

    if not os.path.exists(model_path):
        print(f"Model path does not exist for {model}")
        return None
    model_info["model_path"] = model_path



SUMMARY:
**Function Summary: extract_model_info**

**Purpose and Functionality:**
This function retrieves information about a given machine learning model, including its documentation path and file location.

**Key Implementation Details:**

* Retrieves `model` using the `get_model_doc_path` function to get its documentation path.
* Returns a dictionary containing the following keys:
	+ `model_doc_path`: Path to the model's documentation.
	+ `model_doc_name`: Name of the model's documentation.

**Notable Dependencies and Requirements:**

* `REPO_PATH`: A constant representing the repository path.
* `get_model_doc_path` function.
* External dependencies: This function relies on the existence of the `get_model_doc_path` function, which itself may depend on other external libraries.


key: chroma:document
string_value:
CODE:

def update_relative_imports(filename, model):
    with open(filename, "r") as f:
        filelines = f.read()

    new_file_lines = []
    for line in filelines.split("\n"):
        if line.startswith("from .."):
            new_file_lines.append(line.replace("from ..", "from ..."))
        else:
            new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Summary**

`update_relative_imports()` function updates relative import paths in a Python file. It replaces `..` with `...` to update absolute imports on all lines. The updated file is written back to the original location.

**Key Implementation Details**

* Replaces `from ..` with `from ...` across all lines to convert relative imports to absolute ones.
* Uses `split("\n")` to split the file into individual lines for processing.

**Notable Dependencies/Requirements**

* None explicit dependencies. However, it's assumed that this function runs in a Python environment with necessary modules and libraries available.


key: chroma:document
string_value:
CODE:

def remove_copied_from_statements(model):
    model_path = REPO_PATH / f"src/transformers/models/{model}"
    for file in os.listdir(model_path):
        if file == "__pycache__":
            continue
        file_path = model_path / file
        with open(file_path, "r") as f:
            file_lines = f.read()

        new_file_lines = []
        for line in file_lines.split("\n"):
            if "# Copied from" in line:
                continue
            new_file_lines.append(line)

        with open(file_path, "w") as f:


SUMMARY:
**Summary**

This function, `remove_copied_from_statements`, removes commented lines starting with "# Copied from" from the specified model files. The main purpose is to clean upTransformer models.

**Key Implementation Details:**

* Iterates over lines in each file in the model directory, skipping lines containing "# Copied from".
* Uses `os` and `re` modules implicitly for listing directory contents and file reading/writing.
* Writes modified line-by-line to new file, omitting lines with the specified comment.

**Notable Dependencies/Requirements:**

* `REPO_PATH`: reference path (relative to current working directory) containing model files.
* `os`: file system operations.
* `re`: regular expression matching used implicitly for pattern searching.


key: chroma:document
string_value:
CODE:

def move_model_files_to_deprecated(model):
    model_path = REPO_PATH / f"src/transformers/models/{model}"
    deprecated_model_path = REPO_PATH / f"src/transformers/models/deprecated/{model}"

    if not os.path.exists(deprecated_model_path):
        os.makedirs(deprecated_model_path)

    for file in os.listdir(model_path):
        if file == "__pycache__":
            continue
        repo.git.mv(f"{model_path}/{file}", f"{deprecated_model_path}/{file}")

        # For deprecated files, we then need to update the relative imports


SUMMARY:
**Summary: Migrate Model Files to Deprecation**

### Purpose and Functionality

Migrates model files from `src/transformers/models/{model}` to `src/transformers/models/deprecated/{model}`.

### Key Implementation Details

* Uses `repo.git.mv` to rename files, updating them at the new location.
* Calls `update_relative_imports` to update relative imports for deprecated files.

### Notable Dependencies or Requirements

* `REPO_PATH`: Base path of the repository
* `os.path.exists`, `os.makedirs`, `repo.git.mv`
* `update_relative_imports`

Note: The exact implementation details may vary based on the `update_relative_imports` function, which is not shown in this snippet.


key: chroma:document
string_value:
CODE:

def delete_model_tests(model):
    tests_path = REPO_PATH / f"tests/models/{model}"

    if os.path.exists(tests_path):


SUMMARY:
**Function Summary: `delete_model_tests`**

 Deletes model test files from the repository, removing related Git tracking.

**Main Purpose and Functionality:**
The primary purpose is to delete model test files at the specified path if they exist. It utilizes the Git command-line interface (`repo.git.rm`) to remove these files and their associated Git tracking.

**Key Implementation Details:**

* Uses `os.path.exists` to check for file existence.
* Utilizes `REPO_PATH / f"tests/models/{model}"` as the test path, referencing the `REPO_PATH` constant.
* Executes `repo.git.rm("-r", tests_path)` to remove the files and their tracking using Git.

**Notable Dependencies or Requirements:**

* References `repo`, an existing Git repository object.
* Utilizes external tool, `os`.
* `REPO_PATH`, a constant referencing the repository path.


key: chroma:document
string_value:
CODE:

def get_line_indent(s):


SUMMARY:
**Summary:**

Calculates the line indentation by subtracting the length of the stripped leading spaces from the total string length.

**Key Implementations:**

* `lstrip()` method is used to remove trailing whitespace characters
* Function takes a single argument `s` (string)

**Notable Dependencies/Requirements:**

* None mentioned

This function is used to calculate and return the indentation line in a given string. The implementation relies on the built-in `lstrip()` method, providing a concise and efficient solution for checking leading whitespace characters.


key: chroma:document
string_value:
CODE:

def update_main_init_file(models):
    """
    Replace all instances of model.model_name with model.deprecated.model_name in the __init__.py file

    Args:
        models (List[str]): The models to mark as deprecated
    """
    filename = REPO_PATH / "src/transformers/__init__.py"
    with open(filename, "r") as f:
        init_file = f.read()

    # 1. For each model, find all the instances of model.model_name and replace with model.deprecated.model_name
    for model in models:
        init_file = init_file.replace(f'models.{model}"', f'models.deprecated.{model}"')
        init_file = init_file.replace(f"models.{model} import", f"models.deprecated.{model} import")

    with open(filename, "w") as f:
        f.write(init_file)

    # 2. Resort the imports


SUMMARY:
**Function Summary: `update_main_init_file`**

**Primary Purpose:** Replaces deprecated model names in `__init__.py` file with new, deprecated versions.

**Key Implementation Details:**

* Replaces instances of `model.model_name` with `model.deprecated.model_name` in the `__init__.py` file.
* Also updates import statements to use the deprecated module name.

**Notable Dependencies and Requirements:**

* `REPO_PATH`: The absolute path to the repository's source directory (`"src/transformers/__init__.py"`).
* Uses an external function `sort_imports_in_all_inits(check_only=False)` (not shown in the provided code snippet).


key: chroma:document
string_value:
CODE:

def remove_model_references_from_file(filename, models, condition):
    """
    Remove all references to the given models from the given file

    Args:
        filename (str): The file to remove the references from
        models (List[str]): The models to remove
        condition (Callable): A function that takes the line and model and returns True if the line should be removed
    """
    filename = REPO_PATH / filename
    with open(filename, "r") as f:
        init_file = f.read()

    new_file_lines = []
    for i, line in enumerate(init_file.split("\n")):
        if any(condition(line, model) for model in models):
            continue
        new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Summary:**

`remove_model_references_from_file` function:

* Purpose: Removes references to specified models from a given file by applying a filter condition.
* Key Implementation Details:
	+ Reads the entire file into memory using `init_file = f.read()`.
	+ Iterates over each line, skipping those that match any model based on the `condition` function.
	+ Writes the new filtered file contents back to disk.
* Notable Dependencies and Requirements:
	+ Uses Python's built-in `open()` and `read/write` functions for file input/output.
	+ Assumes a `REPO_PATH` constant or variable defining the repository root path.
	+ Accepts a `condition` function with signature `Callable[[str, str], bool]`.

**RAG (Reusable Anonymous Code) System:**

```markdown
# remove_model_references_from_file
### Purpose
Removes references to specified models from a given file.

### Key Implementation Details

* Reads entire file into memory.
* Applies filter condition to line-model pairs.
* Writes new filtered file contents back to disk.

### Dependencies and Requirements

* Python built-in `open` and `read/write`.
* `REPO_PATH` constant or variable.
* `condition` function signature: `Callable[[str, str], bool]`.
```


key: chroma:document
string_value:
CODE:

def remove_model_config_classes_from_config_check(model_config_classes):
    """
    Remove the deprecated model config classes from the check_config_attributes.py file

    Args:
        model_config_classes (List[str]): The model config classes to remove e.g. ["BertConfig", "DistilBertConfig"]
    """
    filename = REPO_PATH / "utils/check_config_attributes.py"
    with open(filename, "r") as f:
        check_config_attributes = f.read()

    # Keep track as we have to delete comment above too
    in_special_cases_to_allow = False
    in_indent = False
    new_file_lines = []

    for line in check_config_attributes.split("\n"):
        indent = get_line_indent(line)
        if (line.strip() == "SPECIAL_CASES_TO_ALLOW = {") or (line.strip() == "SPECIAL_CASES_TO_ALLOW.update("):
            in_special_cases_to_allow = True

        elif in_special_cases_to_allow and indent == 0 and line.strip() in ("}", ")"):
            in_special_cases_to_allow = False

        if in_indent:
            if line.strip().endswith(("]", "],")):
                in_indent = False
            continue

        if in_special_cases_to_allow and any(
            model_config_class in line for model_config_class in model_config_classes
        ):
            # Remove comments above the model config class to remove
            while new_file_lines[-1].strip().startswith("#"):
                new_file_lines.pop()

            if line.strip().endswith("["):
                in_indent = True

            continue

        elif any(model_config_class in line for model_config_class in model_config_classes):
            continue

        new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Function Summary:**

`remove_model_config_classes_from_config_check`

Removes deprecated model configuration classes from the `check_config_attributes.py` file.

**Purpose and Functionality:**

* Reads the contents of the specified Python file (`utils/check_config_attributes.py`)
* Iterates through each line, tracking indentation and special cases
* Removes comments above deprecated model config classes and rewrites the content in a new file

**Key Implementation Details:**

* Uses regex-free text processing to identify lines with model config classes
* Preserves indentation and formatting when rewriting the content
* Handles edge cases such as nested brackets and escaped newline characters

**Dependencies and Interfaces:**

* None, but may utilize external `REPO_PATH` constant in a larger context.
* Does not implement or inherit any specific interfaces.

**Method Signatures:**

* `remove_model_config_classes_from_config_check(model_config_classes)`: takes a list of string model config class names as input


key: chroma:document
string_value:
CODE:

def add_models_to_deprecated_models_in_config_auto(models):
    """
    Add the models to the DEPRECATED_MODELS list in configuration_auto.py and sorts the list
    to be in alphabetical order.
    """
    filepath = REPO_PATH / "src/transformers/models/auto/configuration_auto.py"
    with open(filepath, "r") as f:
        config_auto = f.read()

    new_file_lines = []
    deprecated_models_list = []
    in_deprecated_models = False
    for line in config_auto.split("\n"):
        if line.strip() == "DEPRECATED_MODELS = [":
            in_deprecated_models = True
            new_file_lines.append(line)
        elif in_deprecated_models and line.strip() == "]":
            in_deprecated_models = False
            # Add the new models to deprecated models list
            deprecated_models_list.extend([f'    "{model}", ' for model in models])
            # Sort so they're in alphabetical order in the file
            deprecated_models_list = sorted(deprecated_models_list)
            new_file_lines.extend(deprecated_models_list)
            # Make sure we still have the closing bracket
            new_file_lines.append(line)
        elif in_deprecated_models:
            deprecated_models_list.append(line)
        else:
            new_file_lines.append(line)

    with open(filepath, "w") as f:


SUMMARY:
**Function Summary:**

`add_models_to_deprecated_models_in_config_auto(models)`

Adds the input `models` to the list of deprecated models in `configuration_auto.py`, updating its contents and sorting the list alphabetically.

**Key Implementation Details:**

- Reads `DEPRECATED_MODELS` from `configuration_auto.py`.
- Appends new models to the list in alphabetical order.
- Updates the file with the modified content.

**Notable Dependencies/Requirements:**

- `REPO_PATH`: reference to a repository-specific path ( exact class/function name required)
- External dependency on:
  - `os` module for file system operations
  - `pathlib` module for handling file paths
  - `shlex` module (not explicitly used but possibly implied by the `split()` and `strip()` functions)

**Interface/Import Notes:**

- No explicit import statements are provided, suggesting that necessary modules and classes will be imported within this code.


key: chroma:document
string_value:
CODE:

def deprecate_models(models):
    # Get model info
    skipped_models = []
    models_info = defaultdict(dict)
    for model in models:
        single_model_info = extract_model_info(model)
        if single_model_info is None:
            skipped_models.append(model)
        else:
            models_info[model] = single_model_info

    model_config_classes = []
    for model, model_info in models_info.items():
        if model in CONFIG_MAPPING:
            model_config_classes.append(CONFIG_MAPPING[model].__name__)
        elif model_info["model_doc_name"] in CONFIG_MAPPING:
            model_config_classes.append(CONFIG_MAPPING[model_info["model_doc_name"]].__name__)
        else:
            skipped_models.append(model)
            print(f"Model config class not found for model: {model}")

    # Filter out skipped models
    models = [model for model in models if model not in skipped_models]

    if skipped_models:
        print(f"Skipped models: {skipped_models} as the model doc or model path could not be found.")
    print(f"Models to deprecate: {models}")

    # Remove model config classes from config check
    print("Removing model config classes from config checks")
    remove_model_config_classes_from_config_check(model_config_classes)

    tip_message = build_tip_message(get_last_stable_minor_release())

    for model, model_info in models_info.items():
        print(f"Processing model: {model}")
        # Add the tip message to the model doc page directly underneath the title
        print("Adding tip message to model doc page")
        insert_tip_to_model_doc(model_info["model_doc_path"], tip_message)

        # Remove #Copied from statements from model's files
        print("Removing #Copied from statements from model's files")
        remove_copied_from_statements(model)

        # Move the model file to deprecated: src/transfomers/models/model -> src/transformers/models/deprecated/model
        print("Moving model files to deprecated for model")
        move_model_files_to_deprecated(model)

        # Delete the model tests: tests/models/model
        print("Deleting model tests")
        delete_model_tests(model)

    # # We do the following with all models passed at once to avoid having to re-write the file multiple times
    print("Updating __init__.py file to point to the deprecated models")
    update_main_init_file(models)

    # Remove model references from other files
    print("Removing model references from other files")
    remove_model_references_from_file(
        "src/transformers/models/__init__.py", models, lambda line, model: model == line.strip().strip(",")
    )
    remove_model_references_from_file(
        "utils/slow_documentation_tests.txt", models, lambda line, model: "/" + model + "/" in line
    )
    remove_model_references_from_file("utils/not_doctested.txt", models, lambda line, model: "/" + model + "/" in line)

    # Add models to DEPRECATED_MODELS in the configuration_auto.py
    print("Adding models to DEPRECATED_MODELS in configuration_auto.py")


SUMMARY:
**Function Summary: deprecate_models**

### Purpose and Functionality:

The `deprecate_models` function is designed to deprecate specific models by updating their documentation, removing references, and refactoring their configurations. It iterates through a list of models, extracting information about each model from an `extract_model_info` function call. The function checks if the model has a configured class in the `CONFIG_MAPPING`, adds a tip message to the model's documentation page, removes redundant statements, moves the model files to the deprecated directory, deletes test files, and updates other files that reference the models.

### Key Implementation Details:

*   Iterates through a list of models using `models` as input.
*   Calls `extract_model_info(model)` for each model to gather information.
*   Checks if a configured class exists in `CONFIG_MAPPING` or by name within `CONFIG_MAPPING`.
*   Updates the last stable minor release version using `get_last_stable_minor_release()`.
*   Uses `build_tip_message(tip_message)`, `remove_model_config_classes_from_config_check(model_config_classes)`, and other functions to finalize model depreciation.
*   Iterates over models, processing each one individually using `insert_tip_to_model_doc`, `remove_copied_from_statements`, `move_model_files_to_deprecated`, `delete_model_tests`, and updates files like `update_main_init_file` to point to the deprecated models.

### Dependencies and Requirements:

*   Uses `defaultdict` from Python's standard library for creating default dictionaries.
*   References `CONFIG_MAPPING` (a dictionary of model configurations) at least once during execution, which must be defined elsewhere in the codebase.
*   Calls functions with external dependencies like `remove_model_references_from_file`, `update_main_init_file`, and others that reference files or classes not specified within this function.

### Notable Interfaces:

The function provides a few notable interfaces to other components of the system:
    
*   `CONFIG_MAPPING`: A dictionary mapping model names or documentation lines to their respective configuration classes.
*   External dependencies such as `update_main_init_file`, `remove_model_references_from_file` (referencing specific files or directories), and `build_tip_message`.


key: chroma:document
string_value:
CODE:

def sort_auto_mapping(fname: str, overwrite: bool = False) -> Optional[bool]:
    """
    Sort all auto mappings in a file.

    Args:
        fname (`str`): The name of the file where we want to sort auto-mappings.
        overwrite (`bool`, *optional*, defaults to `False`): Whether or not to fix and overwrite the file.

    Returns:
        `Optional[bool]`: Returns `None` if `overwrite=True`. Otherwise returns `True` if the file has an auto-mapping
        improperly sorted, `False` if the file is okay.
    """
    with open(fname, "r", encoding="utf-8") as f:
        content = f.read()

    lines = content.split("\n")
    new_lines = []
    line_idx = 0
    while line_idx < len(lines):
        if _re_intro_mapping.search(lines[line_idx]) is not None:
            # Start of a new mapping!
            indent = len(re.search(r"^(\s*)\S", lines[line_idx]).groups()[0]) + 8
            while not lines[line_idx].startswith(" " * indent + "("):
                new_lines.append(lines[line_idx])
                line_idx += 1

            blocks = []
            while lines[line_idx].strip() != "]":
                # Blocks either fit in one line or not
                if lines[line_idx].strip() == "(":
                    start_idx = line_idx
                    while not lines[line_idx].startswith(" " * indent + ")"):
                        line_idx += 1
                    blocks.append("\n".join(lines[start_idx : line_idx + 1]))
                else:
                    blocks.append(lines[line_idx])
                line_idx += 1

            # Sort blocks by their identifiers
            blocks = sorted(blocks, key=lambda x: _re_identifier.search(x).groups()[0])
            new_lines += blocks
        else:
            new_lines.append(lines[line_idx])
            line_idx += 1

    if overwrite:
        with open(fname, "w", encoding="utf-8") as f:
            f.write("\n".join(new_lines))
    else:


SUMMARY:
**Function Summary: `sort_auto_mapping`**

The `sort_auto_mapping` function sorts the auto-mappings in a given file.

### Purpose and Functionality:

Automatically rearranges indented lines containing mappings in the specified file to adhere to a specific format. Compares and returns whether the original formatting is preserved upon sorting (when `overwrite=False`) or if changes were made for overwriting (`overwrite=True`).

### Key Implementation Details:

1.  **Regex-based Pattern Matching**:
    *   `_re_intro_mapping.search()`: Identifies the start of a new mapping in lines.
2.  **Blocks and Lines Processing**:
    *   `indentation calculation`: Calculates indentation length before processing each mapping block.
    *   `block decomposition` (`if lines[line_idx].strip() == "(":`): Tackles blocks divided across multiple lines with indentation matching the first line's formatting.
3.  **Sorting Algorithm**:
    *   `blocks sorted by ids` (`sorted(blocks, key=lambda x: _re_identifier.search(x).groups()[0])`): Orders and groups mappings according to their unique identifiers.

### Notable Dependencies or Requirements:

1.  `re` module (regular expressions) for pattern matching.
2.  External file operations via `open` function with UTF-8 encoding.

**API Reference:**

*   `_re_intro_mapping`, `_re_identifier`: Externally defined regular expression patterns used within the sorting logic, not part of this summary but crucial for implementation context.

### Example Usage:
```python
result = sort_auto_mapping('path/to/file.txt')
print(result)  # Returns 'None' if overwrite is False and file has been properly sorted, False otherwise.
```


key: chroma:document
string_value:
CODE:

def sort_all_auto_mappings(overwrite: bool = False):
    """
    Sort all auto mappings in the library.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`): Whether or not to fix and overwrite the file.
    """
    fnames = [os.path.join(PATH_TO_AUTO_MODULE, f) for f in os.listdir(PATH_TO_AUTO_MODULE) if f.endswith(".py")]
    diffs = [sort_auto_mapping(fname, overwrite=overwrite) for fname in fnames]

    if not overwrite and any(diffs):
        failures = [f for f, d in zip(fnames, diffs) if d]
        raise ValueError(
            f"The following files have auto mappings that need sorting: {', '.join(failures)}. Run `make style` to fix"
            " this."


SUMMARY:
**Summary**

Function `sort_all_auto_mappings`: Sorts all auto mappings in the library, fixing inconsistencies if required. 

### Purpose
Synchronizes Python file auto mappings within a module.

### Implementation Details

- Fetches relevant files using list comprehension with path manipulation (`os.path.join`).
- Iterates through each file, calls `sort_auto_mapping` to sort its mappings.
- Checks for non-overwritten changes after sorting; raises an error if updates are detected.


key: chroma:document
string_value:
CODE:

class CustomTokenizerFast(BertTokenizerFast):
    slow_tokenizer_class = CustomTokenizer


SUMMARY:
**Summary:**

This class `CustomTokenizerFast` inherits from `BertTokenizerFast`. It overrides the default tokenizer by specifying its own tokenizer class as `SlowTokenizer`. The primary purpose is to enable the use of a custom, slow tokenization mechanism in place of the FastTokenizer.

**Key Implementation Details:**

* Inherited Class: `BertTokenizerFast`
* Implemented Class: `CustomTokenizer` (specifically assigned as `slow_tokenizer_class`)
* No additional functionality or method implementations provided; primarily serves as an override for the default tokenizer configuration.


key: chroma:document
string_value:
CODE:

class CustomTokenizer(BertTokenizer):


SUMMARY:
**Summary**

Class: `CustomTokenizer`
Inherits from: `BertTokenizer`

**Purpose and Functionality**: Creates a custom version of the BERT tokenizer.

**Key Implementation Details**:

* No specific tokenization algorithms or patterns are defined, inheriting functionality from `BertTokenizer`.
* No method signatures or interfaces are overridden by default.
 

**Notable Dependencies/Requirements**:
* Inherits from: `BertTokenizer`
* External dependencies not specified.

No key implementation details or interface implementations to note.


key: chroma:document
string_value:
CODE:

class CustomModel(PreTrainedModel):
    config_class = CustomConfig

    def __init__(self, config):
        super().__init__(config)
        self.linear = torch.nn.Linear(config.hidden_size, config.hidden_size)

    def forward(self, x):
        return self.linear(x)

    def _init_weights(self, module):


SUMMARY:
**Summary:**

The `CustomModel` class is a custom PyTorch model derived from `PreTrainedModel`. It uses the `CustomConfig` configuration.

**Main Purpose and Functionality:**
It defines a basic neural network architecture with one linear layer, where the input `x` passes through this layer to produce an output.

**Key Implementation Details:**

*   A linear layer (`self.linear`) with input/ouput size of `config.hidden_size`.
*   The `forward` method applies the linear transformation to the input `x`.

**Notable Dependencies and Requirements:**

*   Import from PyTorch (`torch.nn.Linear`).
*   Inheritance from `PreTrainedModel`.


key: chroma:document
string_value:
CODE:

class NoSuperInitModel(PreTrainedModel):
    config_class = NoSuperInitConfig

    def __init__(self, config):
        super().__init__(config)
        self.linear = torch.nn.Linear(config.attribute, config.attribute)

    def forward(self, x):
        return self.linear(x)

    def _init_weights(self, module):


SUMMARY:
**Summary:**

The `NoSuperInitModel` class is a PyTorch-based pre-trained model that extends the `PreTrainedModel` class. It defines:

* A linear layer with a configuration attribute `config.attribute`.
* A forward pass through the linear layer, mapping input `x` to its output.
* An `_init_weights` method that sets no weights (implies super-init behavior).

**Key Details:**

* **Linear Layer**: `self.linear = torch.nn.Linear(config.attribute, config.attribute)`
* **Forward Pass**: `return self.linear(x)`
* **_init_weights Method**: Implicitly inherits from superclass's behavior

**Dependencies/Requirements:**

* PyTorch (`torch`)
* `PreTrainedModel` class (inherited)
* `NoSuperInitConfig` configuration class ( referenced)


key: chroma:document
string_value:
CODE:

class CustomImageProcessor(CLIPImageProcessor):


SUMMARY:
**Code Summary:**

*   Class Name: `CustomImageProcessor`
*   Inheritance: Extends `CLIPImageProcessor` to create a custom image processor.

**Additional Details:**

*   No specific implementation details are provided, as the class solely inherits from `CLIPImageProcessor`.


key: chroma:document
string_value:
CODE:

def softmax(outputs):
    maxes = np.max(outputs, axis=-1, keepdims=True)
    shifted_exp = np.exp(outputs - maxes)


SUMMARY:
**Concise Summary**

This function implements the softmax activation function. It takes an input array `outputs` of shape `(num_classes, batch_size)` and returns a normalized value for each class.

The primary purpose is to prevent output values from growing exponentially large by normalizing them over their maximum value across classes.

**Key Implementation Details:**

- Uses NumPy's vectorized operations to calculate the softmax output.
- Employs shift-invariant exponentiation (`outputs - maxes`) to mitigate numerical instability issues.

**Notable Dependencies and Requirements:**

* `numpy` (imported as `np` for array operations)
* `outputs` input should be an array of shape `(num_classes, batch_size)` containing output values.


key: chroma:document
string_value:
CODE:

class PairClassificationPipeline(Pipeline):
    def _sanitize_parameters(self, **kwargs):
        preprocess_kwargs = {}
        if "second_text" in kwargs:
            preprocess_kwargs["second_text"] = kwargs["second_text"]
        return preprocess_kwargs, {}, {}

    def preprocess(self, text, second_text=None):
        return self.tokenizer(text, text_pair=second_text, return_tensors=self.framework)

    def _forward(self, model_inputs):
        return self.model(**model_inputs)

    def postprocess(self, model_outputs):
        logits = model_outputs.logits[0].numpy()
        probabilities = softmax(logits)

        best_class = np.argmax(probabilities)
        label = self.model.config.id2label[best_class]
        score = probabilities[best_class].item()
        logits = logits.tolist()


SUMMARY:
**PairClassificationPipeline Class Summary**

* **Purpose:** This class implements a pipeline for pair classification tasks using a pre-trained model. It sanitizes input parameters, preprocesses text inputs with optional second text, and postprocesses model outputs to extract confidence scores and predicted labels.

* **Key Implementation Details:**
	+ `preprocess` method takes in text inputs and an optional second text, and uses the `tokenizer` to generate input tensors in a specific framework.
	+ `_forward` method passes the preprocessed input to the trained model for inference.
	+ `postprocess` method extracts attention scores from the model output, converts them to probabilities using softmax, and determines the predicted label by finding the class with the highest probability.

* **Notable Dependencies:**
	+ `tokenizer`: a tokenizer object used in the `preprocess` method
	+ `model`: a pre-trained model instance that takes in input tensors in the same framework as the pipeline
	+ `self.framework`: the specific framework used for tensor generation (e.g. PyTorch, TensorFlow)
	+ `softmax`, `logits`, and related functions: commonly used mathematical functions for probability calculations


key: chroma:document
string_value:
CODE:

class CustomConfig(PretrainedConfig):
    model_type = "custom"

    def __init__(self, attribute=1, **kwargs):
        self.attribute = attribute


SUMMARY:
**Summary**

`CustomConfig` is a custom configuration class for the `PretrainedConfig` base class. It defines a model type as "custom" and extends initialization from its parent with an additional `attribute` parameter, defaulting to 1.

**Key Implementation Details:**

* Customizes model type to `model_type = "custom"`
* Extends __init__ from `PretrainedConfig` with `attribute` parameter

**Notable Dependencies or Requirements:**
None explicitly listed. Inherited interfaces: `PretrainedConfig`.


key: chroma:document
string_value:
CODE:

class NoSuperInitConfig(PretrainedConfig):
    model_type = "custom"

    def __init__(self, attribute=1, **kwargs):


SUMMARY:
**Summary**

This `NoSuperInitConfig` class inherits from `PretrainedConfig`, defining a custom configuration for a specific model. The primary purpose is to initialize the configuration with an optional `attribute` parameter.

**Key Details**

* Inherited from: `PretrainedConfig`
* Model type: "custom"
* Initialization:
	+ Required attribute: `attribute=1` (default value)

**Notable Dependencies/Requirements**

* None explicitly listed, but assumes access to `PretrainedConfig` class and related dependencies.


key: chroma:document
string_value:
CODE:

class CustomProcessor(ProcessorMixin):
    feature_extractor_class = "AutoFeatureExtractor"


SUMMARY:
**Summary:**

This `CustomProcessor` class inherits from `ProcessorMixin` and defines two key attributes:

* `feature_extractor_class`: specifies the use of the `AutoFeatureExtractor` component.
* `tokenizer_class`: specifies the use of the `AutoTokenizer` component.

No additional implementation details or dependencies are defined for this class.


key: chroma:document
string_value:
CODE:

class CustomFeatureExtractor(Wav2Vec2FeatureExtractor):


SUMMARY:
**Summary**

A custom `CustomFeatureExtractor` class that inherits from the `Wav2Vec2FeatureExtractor` class, effectively extending the functionality of the parent class with no additional implementation.

**Key Implementation Details**

* Inherited from `Wav2Vec2FeatureExtractor`, implying an extension to the Wav2Vec2 model's feature extraction capabilities.
* No additional methods or attributes are defined, suggesting a focus on leveraging the existing `Wav2Vec2FeatureExtractor` functionality with no new additions.

**Notable Dependencies**

* `Wav2Vec2FeatureExtractor`: The parent class from which `CustomFeatureExtractor` inherits.

**Additional Requirements**

None.


key: chroma:document
string_value:
CODE:

def pytest_configure(config):
    config.addinivalue_line(
        "markers", "is_pt_tf_cross_test: mark test to run only when PT and TF interactions are tested"
    )
    config.addinivalue_line(
        "markers", "is_pt_flax_cross_test: mark test to run only when PT and FLAX interactions are tested"
    )
    config.addinivalue_line("markers", "is_pipeline_test: mark test to run only when pipelines are tested")
    config.addinivalue_line("markers", "is_staging_test: mark test to run only in the staging environment")
    config.addinivalue_line("markers", "accelerate_tests: mark test that require accelerate")
    config.addinivalue_line("markers", "agent_tests: mark the agent tests that are run on their specific schedule")


SUMMARY:
**RAG Code Summary**

Function Name: `pytest_configure`

Purpose:
Enable and define custom markers for pytest tests in the configuration.

Key Implementation Details:

* Add custom marker lines with descriptive names, e.g. `is_pt_tf_cross_test`, `accelerate_tests`
* Define a total of 7 unique marker settings

Notable Dependencies or Requirements:
- Pytest library for testing framework
- N/A (only internal pytest configuration)

Methods and Interfaces Used:

None

External Dependencies:
- None

Design Choices:

- Utilizing `config.addinivalue_line` to extend pytest configuration
- Employing custom markers for specific testing scenarios

Primary Purpose: Enhance pytest test execution by providing customizable testing flag configurations.


key: chroma:document
string_value:
CODE:

def pytest_collection_modifyitems(items):
    for item in items:
        if any(test_name in item.nodeid for test_name in NOT_DEVICE_TESTS):


SUMMARY:
**Summary:**

`pytest_collection_modifyitems` is a function that modifies a list of test items. It filters out items whose names contain specific device-related tests (`NOT_DEVICE Tests`) and adds the `not_device_test` marker to remaining items.

**Key Details:**

* Filters test items by name using `any(check in item.nodeid for check in NOT_DEVICE_TESTS)`
* Adds `pytest.mark.not_device_test` marker to filtered items
* No specific algorithm or pattern used, but relies on Python's built-in string matching logic

**Implementation Dependencies:**

* `pytest`

**Interfaces/Inheritance:**

* Implements the ` pytest.Collection` interface


key: chroma:document
string_value:
CODE:

def pytest_addoption(parser):
    from transformers.testing_utils import pytest_addoption_shared



SUMMARY:
**Function Summary**

`pytest_addoption`: A custom pytest hook to add options for testing Hugging Face model transformers.

**Key Implementation Details:**

* Calls `pytest_addoption_shared` from `transformers.testing_utils.pytest` with the same parser instance.

**Notable Dependencies/Requirements:**

* `pytest.testing_utils`: Provides a shared implementation for adding options in pytest.
* No external dependencies specified.


key: chroma:document
string_value:
CODE:

def pytest_terminal_summary(terminalreporter):
    from transformers.testing_utils import pytest_terminal_summary_main

    make_reports = terminalreporter.config.getoption("--make-reports")
    if make_reports:


SUMMARY:
**Summary: Pytest Terminal Summary Function**

The `pytest_terminal_summary` function is a custom pytest fixture that determines whether to generate terminal reports. When the "--make-reports" option is specified, it calls the `pytest_terminal_summary_main` function with the provided reporter.

**Key Details:**

* Primary purpose: Determine whether to generate terminal reports based on the "--make-reports" option.
* Implementation: Checks for the existence of the "--make-reports" option and uses it as input for the `pytest_terminal_summary_main` function.

**Dependencies/Notable Requirements:**

* `terminalreporter`: The input reporter used by pytest-terminal-summary.
* `pytransformers`: A library containing the `pytest_terminal_summary_main` function.


key: chroma:document
string_value:
CODE:

def pytest_sessionfinish(session, exitstatus):
    # If no tests are collected, pytest exists with code 5, which makes the CI fail.
    if exitstatus == 5:


SUMMARY:
**Function Summary:**

`pytest_sessionfinish`: A fixture function executed when all tests in a Pytest session have finished execution.

**Primary Purpose:** Prevents Pytest from reporting an error code of 5, indicating no tests were collected, and instead sets the exit status to successful (0).

**Key Implementation Details:**

* Sets `session.exitstatus` to 0 if the current test session has an unknown or undefined existence code of 5 (`exitstatus == 5`)

**Notable Dependencies/Requirements:**

* Pytest framework
* `pytest ExitStatusError` exception


key: chroma:document
string_value:
CODE:

class CustomOutputChecker(OutputChecker):
    def check_output(self, want, got, optionflags):
        if IGNORE_RESULT & optionflags:
            return True


SUMMARY:
**Summary**

### Class Overview

`CustomOutputChecker` extends the `OutputChecker` class and overrides its `check_output` method.

### Primary Purpose

Customizes the behavior of output checking based on specific option flags (`optionflags`).

### Key Implementation Details

* Uses bitwise operator to check for presence of `IGNORE_RESULT` flag in `optionflags`, returning True if it's present.
* Calls the inherited `OutputChecker.check_output` method with the same arguments.

### Notable Dependencies/Requirements

* Extends `OutputChecker` class
* Imports `OutputChecker` and uses its `check_output` method directly.


key: docstring
string_value:
# since we save this data in src/transformers/dependency_versions_table.py it can be easily accessed from
# anywhere. If you need to quickly access the data from this table in a shell, you can do so easily with:
#
# python -c 'import sys; from transformers.dependency_versions_table import deps; \
# print(" ".join([ deps[x] for x in sys.argv[1:]]))' tokenizers datasets
#
# Just pass the desired package names to that script as it's shown with 2 packages above.
#
# If transformers is not yet installed and the work is done from the cloned repo remember to add `PYTHONPATH=src` to the script above
#
# You can then feed this for example to `pip`:
#
# pip install -U $(python -c 'import sys; from transformers.dependency_versions_table import deps; \
# print(" ".join([deps[x] for x in sys.argv[1:]]))' tokenizers datasets)
#


key: chroma:document
string_value:
CODE:

def deps_list(*pkgs):


SUMMARY:
**Function Summary**

`deps_list` function generates a list of dependencies for a given set of packages (`*pkgs`) by retrieving the dependency information from `dependency_versions_table.py`.

**Key Implementation Details**

* List comprehension to generate dependency dependencies in `[deps[pkg] for pkg in pkgs]`.
* External dependency or interface: none.
* Notable design choices:

  * Stores data in a file (`dependency_versions_table.py`) for easy access.
  * Allows external execution using Python shell script.

**Primary Purpose**

Generates list of dependencies for specified packages.


key: chroma:document
string_value:
CODE:

class DepsTableUpdateCommand(Command):
    """
    A custom distutils command that updates the dependency table.
    usage: python setup.py deps_table_update
    """

    description = "build runtime dependency table"
    user_options = [
        # format: (long option, short option, description).
        ("dep-table-update", None, "updates src/transformers/dependency_versions_table.py"),
    ]

    def initialize_options(self):
        pass

    def finalize_options(self):
        pass

    def run(self):
        entries = "\n".join([f'    "{k}": "{v}",' for k, v in deps.items()])
        content = [
            "# THIS FILE HAS BEEN AUTOGENERATED. To update:",
            "# 1. modify the `_deps` dict in setup.py",
            "# 2. run `make deps_table_update``",
            "deps = {",
            entries,
            "}",
            "",
        ]
        target = "src/transformers/dependency_versions_table.py"
        print(f"updating {target}")
        with open(target, "w", encoding="utf-8", newline="\n") as f:


SUMMARY:
**Summary:**

The `DepsTableUpdateCommand` class is a custom distutils command that updates the dependency table. It parses the `_deps` dictionary and generates a runtime dependency table by creating a new file `src/transformers/dependency_versions_table.py`.

**Key Implementation Details:**

* Generates content for a Python file at `src/transformers/dependency_versions_table.py`
* Updates the file with pre-defined header comment
* Writes the updated `_deps` dictionary to the file

**Notable Dependencies and Requirements:**

* None explicit dependencies listed
* Implicit dependency on `setup.py` and `make` utility

**Method Signatures and Interfaces:**

* `initialize_options()`: an empty method used for initialization purposes
* `finalize_options()`: an empty method used for finalization purposes
* `run()`: the main entry point that generates the content for the file


key: chroma:document
string_value:
CODE:

def parse_pytest_output(file_path):
    skipped_tests = {}
    skipped_count = 0
    with open(file_path, 'r') as file:
        for line in file:
            match = re.match(r'^SKIPPED \[(\d+)\] (tests/.*): (.*)$', line)
            if match:
                skipped_count += 1
                test_file, test_line, reason = match.groups()
                skipped_tests[reason] = skipped_tests.get(reason, []) + [(test_file, test_line)]
    for k,v in sorted(skipped_tests.items(), key=lambda x:len(x[1])):
        print(f"{len(v):4} skipped because: {k}")


SUMMARY:
**Summary**

The `parse_pytest_output` function parses the output of a Pytest test run from a files specified by `file_path`. It extracts skipped tests, counting and printing their reasons, along with the number of skipped tests.

**Main Purpose and Functionality**

*   Extracts skipped tests from a Pytest output file
*   Counts and prints each reason for skipping along with its corresponding test files and lines

**Key Implementation Details**

*   Uses regular expressions to match `SKIPPED` test markers (`'^SKIPPED \[(\d+)\] (tests/.*): (.*)$'`)
*   Iterates through the Pytest output file, processing each line for skipped tests
*   Uses a dictionary (`skipped_tests`) to store skipping reasons as keys and lists of corresponding tests as values

**Notable Dependencies or Requirements**

*   Utilizes `re` module for regular expression matching
*   Implies usage of the `sorted()` function for grouping sorted skips by reason length


key: chroma:document
string_value:
CODE:

def parse_pytest_failure_output(file_path):
    failed_tests = {}
    failed_count = 0
    with open(file_path, 'r') as file:
        for line in file:
            match = re.match(r'^FAILED (tests/.*) - (.*): (.*)$', line)
            if match:
                failed_count += 1
                _, error, reason = match.groups()
                failed_tests[reason] = failed_tests.get(reason, []) + [error]
    for k,v in sorted(failed_tests.items(), key=lambda x:len(x[1])):
        print(f"{len(v):4} failed because `{v[0]}` -> {k}")
    print("Number of failed tests:", failed_count)
    if failed_count>0:


SUMMARY:
**Function Summary: `parse_pytest_failure_output`**

**Purpose and Functionality:**
This function parses the output of pytest failure reports from a text file, extracting test cases and their corresponding error messages.

**Key Implementation Details:**

* Uses regular expression to match failure report lines, which follow the format: 'FAILED (tests/<test_name>): (error message); (reason for failure)'
* Iterates through the file, incrementing a count of failed tests and storing error messages in a dictionary (`failed_tests`) keyed by reason.
* Prints the top reasons with the most errors, alongside their frequencies and corresponding error messages.

**Notable Dependencies or Requirements:**

* `re` (regular expression) module for pattern matching
* No other external dependencies explicitly mentioned

Note: The function exits if any tests failed (return code 1).


key: chroma:document
string_value:
CODE:

def parse_pytest_errors_output(file_path):
    print(file_path)
    error_tests = {}
    error_count = 0
    with open(file_path, 'r') as file:
        for line in file:
            match = re.match(r'^ERROR (tests/.*) - (.*): (.*)$', line)
            if match:
                error_count += 1
                _, test_error, reason = match.groups()
                error_tests[reason] = error_tests.get(reason, []) + [test_error]
    for k,v in sorted(error_tests.items(), key=lambda x:len(x[1])):
        print(f"{len(v):4} errored out because of `{v[0]}` -> {k}")
    print("Number of errors:", error_count)
    if error_count>0:


SUMMARY:
**Function Summary: `parse_pytest_errors_output`**

### Main Purpose and Functionality

This function parses the output of pytest to collect error test cases. It reads a file at `file_path`, extracts error test cases from lines matching the `ERROR` pattern, and prints the error frequency along with reasons and corresponding test errors.

**Key Implementation Details**

* Uses regular expression (`re.match`) to extract error message components.
* Groups extracted fields into test errors for analysis (e.g., `error_tests[reason] = ...`).
* Sorts test errors alphabetically by reason before printing results.

**Notable Dependencies or Requirements**

* `re` module for regular expressions
* External file at `file_path` containing pytest output


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--file", help="file to parse")
    parser.add_argument("--skip", action="store_true", help="show skipped reasons")
    parser.add_argument("--fail", action="store_true", help="show failed tests")
    parser.add_argument("--errors", action="store_true", help="show failed tests")
    args = parser.parse_args()

    if args.skip:
        parse_pytest_output(args.file)

    if args.fail:
        parse_pytest_failure_output(args.file)

    if args.errors:


SUMMARY:
**Summary**

This `main` function defines an argument parser to handle parsing file options. It takes the following arguments:

* `--file`: input file to be parsed
* `--skip`: show skipped reasons
* `--fail`: show failed tests
* `--errors`: show parsing errors (not `--fail`, as in different terminology)

The function parses the command-line arguments and then calls separate functions `parse_pytest_output`, `parse_pytest_failure_output`, and `parse_pytest_errors_output` based on the input parameters. These functions are not shown in this code snippet.

Key implementation details:

* Uses Argparse to define a parser with file-specific arguments
* Parses command-line arguments using `parser.parse_args()`
* Calls separate parsing functions based on input parameters

Notable dependencies or requirements:

* Argparse library for parsing command-line arguments (`argparse.ArgumentParser`)
* Custom functions: `parse_pytest_output`, `parse_pytest_failure_output`, and `parse_pytest_errors_output` (not shown)


key: chroma:document
string_value:
CODE:

class EmptyJob:
    job_name = "empty"

    def to_dict(self):
        steps = [{"run": 'ls -la'}]
        if self.job_name == "collection_job":
            steps.extend(
                [
                    "checkout",
                    {"run": "pip install requests || true"},
                    {"run": """while [[ $(curl --location --request GET "https://circleci.com/api/v2/workflow/$CIRCLE_WORKFLOW_ID/job" --header "Circle-Token: $CCI_TOKEN"| jq -r '.items[]|select(.name != "collection_job")|.status' | grep -c "running") -gt 0 ]]; do sleep 5; done || true"""},
                    {"run": 'python utils/process_circleci_workflow_test_reports.py --workflow_id $CIRCLE_WORKFLOW_ID || true'},
                    {"store_artifacts": {"path": "outputs"}},
                    {"run": 'echo "All required jobs have now completed"'},
                ]
            )

        return {
            "docker": copy.deepcopy(DEFAULT_DOCKER_IMAGE),
            "resource_class": "small",
            "steps": steps,


SUMMARY:
**Summary**

The `EmptyJob` class creates a CI/CD workflow job. Its primary purpose is to simulate a collection job with multiple parallelized steps.

**Key Implementation Details:**

*   **Steps and Script:** The job has two main execution paths:
    *   A simple "ls -la" command.
    *   A more complex script that includes:
        *   Checking if the specified CircleCI job is running.
        *   Installing required packages using pip (requests).
        *   Running a Python script to process test reports.
    This script requires external dependencies, including `circleci.com` and `requests`.

**Dependencies:**

*   External dependency on `circleci.com` for API access:
    +   `$CIRCLE_WORKFLOW_ID`
    +   `$CCI_TOKEN`
*   External dependency on pip for package installation (`requests`)
*   Custom class `utils/process_circleci_workflow_test_reports.py` for processing test reports

**Notable Features:**

*   Modular execution using parallelized steps with conditional logic
*   Involves external interactions with CircleCI API and pip package manager


key: chroma:document
string_value:
CODE:

def create_circleci_config(folder=None):
    if folder is None:
        folder = os.getcwd()
    os.environ["test_preparation_dir"] = folder
    jobs = [k for k in ALL_TESTS if os.path.isfile(os.path.join("test_preparation" , f"{k.job_name}_test_list.txt") )]
    print("The following jobs will be run ", jobs)

    if len(jobs) == 0:
        jobs = [EmptyJob()]
    else:
        print("Full list of job name inputs", {j.job_name + "_test_list":{"type":"string", "default":''} for j in jobs})
        # Add a job waiting all the test jobs and aggregate their test summary files at the end
        collection_job = EmptyJob()
        collection_job.job_name = "collection_job"
        jobs = [collection_job] + jobs

    config = {
        "version": "2.1",
        "parameters": {
            # Only used to accept the parameters from the trigger
            "nightly": {"type": "boolean", "default": False},
            "tests_to_run": {"type": "string", "default": ''},
            **{j.job_name + "_test_list":{"type":"string", "default":''} for j in jobs},
            **{j.job_name + "_parallelism":{"type":"integer", "default":1} for j in jobs},
        },
        "jobs": {j.job_name: j.to_dict() for j in jobs}
    }
    if "CIRCLE_TOKEN" in os.environ:
        # For private forked repo. (e.g. new model addition)
        config["workflows"] = {"version": 2, "run_tests": {"jobs": [{j.job_name: {"context": ["TRANSFORMERS_CONTEXT"]}} for j in jobs]}}
    else:
        # For public repo. (e.g. `transformers`)
        config["workflows"] = {"version": 2, "run_tests": {"jobs": [j.job_name for j in jobs]}}
    with open(os.path.join(folder, "generated_config.yml"), "w") as f:


SUMMARY:
**Function Summary**

The `create_circleci_config` function generates a CI configuration file for CircleCI. It takes an optional `folder` parameter, defaulting to the current working directory. The function constructs a configuration object with parameters and jobs, which is then written to a YAML file.

**Key Implementation Details**

* Iterates over `ALL_TESTS` job names and selects only those with corresponding test files.
* Adds a "collection_job" that waits for all other jobs to finish.
* Constructs the `config` dictionary with:
	+ Parameters for nightly builds and test summaries.
	+ Jobs as dictionaries, populated from their respective `to_dict()` methods.

**Notable Dependencies**

* `os`, `ALL_TESTS` (custom data structure containing job names).
* `yaml`.
* `CircleCI` configuration file formatting conventions (~ pipeline transformations).

**Interfaces and Inherited Dependencies**

* None explicit dependencies.
* Assumes access to `ALL_TESTS` and CircleCI-specific formatting conventions.


key: chroma:document
string_value:
CODE:

class ModelCard:
    r"""
    Structured Model Card class. Store model card as well as methods for loading/downloading/saving model cards.

    Please read the following paper for details and explanation on the sections: "Model Cards for Model Reporting" by
    Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer,
    Inioluwa Deborah Raji and Timnit Gebru for the proposal behind model cards. Link: https://arxiv.org/abs/1810.03993

    Note: A model card can be loaded and saved to disk.
    """

    def __init__(self, **kwargs):
        warnings.warn(
            "The class `ModelCard` is deprecated and will be removed in version 5 of Transformers", FutureWarning
        )
        # Recommended attributes from https://arxiv.org/abs/1810.03993 (see papers)
        self.model_details = kwargs.pop("model_details", {})
        self.intended_use = kwargs.pop("intended_use", {})
        self.factors = kwargs.pop("factors", {})
        self.metrics = kwargs.pop("metrics", {})
        self.evaluation_data = kwargs.pop("evaluation_data", {})
        self.training_data = kwargs.pop("training_data", {})
        self.quantitative_analyses = kwargs.pop("quantitative_analyses", {})
        self.ethical_considerations = kwargs.pop("ethical_considerations", {})
        self.caveats_and_recommendations = kwargs.pop("caveats_and_recommendations", {})

        # Open additional attributes
        for key, value in kwargs.items():
            try:
                setattr(self, key, value)
            except AttributeError as err:
                logger.error(f"Can't set {key} with value {value} for {self}")
                raise err

    def save_pretrained(self, save_directory_or_file):
        """Save a model card object to the directory or file `save_directory_or_file`."""
        if os.path.isdir(save_directory_or_file):
            # If we save using the predefined names, we can load using `from_pretrained`
            output_model_card_file = os.path.join(save_directory_or_file, MODEL_CARD_NAME)
        else:
            output_model_card_file = save_directory_or_file

        self.to_json_file(output_model_card_file)
        logger.info(f"Model card saved in {output_model_card_file}")

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
        r"""
        Instantiate a [`ModelCard`] from a pre-trained model model card.

        Parameters:
            pretrained_model_name_or_path: either:

                - a string, the *model id* of a pretrained model card hosted inside a model repo on huggingface.co.
                - a path to a *directory* containing a model card file saved using the [`~ModelCard.save_pretrained`]
                  method, e.g.: `./my_model_directory/`.
                - a path or url to a saved model card JSON *file*, e.g.: `./my_model_directory/modelcard.json`.

            cache_dir: (*optional*) string:
                Path to a directory in which a downloaded pre-trained model card should be cached if the standard cache
                should not be used.

            kwargs: (*optional*) dict: key/value pairs with which to update the ModelCard object after loading.

                - The values in kwargs of any keys which are model card attributes will be used to override the loaded
                  values.
                - Behavior concerning key/value pairs whose keys are *not* model card attributes is controlled by the
                  *return_unused_kwargs* keyword parameter.

            proxies: (*optional*) dict, default None:
                A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128',
                'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.

            return_unused_kwargs: (*optional*) bool:

                - If False, then this function returns just the final model card object.
                - If True, then this functions returns a tuple *(model card, unused_kwargs)* where *unused_kwargs* is a
                  dictionary consisting of the key/value pairs whose keys are not model card attributes: ie the part of
                  kwargs which has not been used to update *ModelCard* and is otherwise ignored.

        Examples:

        ```python
        # Download model card from huggingface.co and cache.
        modelcard = ModelCard.from_pretrained("google-bert/bert-base-uncased")
        # Model card was saved using *save_pretrained('./test/saved_model/')*
        modelcard = ModelCard.from_pretrained("./test/saved_model/")
        modelcard = ModelCard.from_pretrained("./test/saved_model/modelcard.json")
        modelcard = ModelCard.from_pretrained("google-bert/bert-base-uncased", output_attentions=True, foo=False)
        ```"""
        cache_dir = kwargs.pop("cache_dir", None)
        proxies = kwargs.pop("proxies", None)
        return_unused_kwargs = kwargs.pop("return_unused_kwargs", False)
        from_pipeline = kwargs.pop("_from_pipeline", None)

        user_agent = {"file_type": "model_card"}
        if from_pipeline is not None:
            user_agent["using_pipeline"] = from_pipeline

        is_local = os.path.isdir(pretrained_model_name_or_path)
        if os.path.isfile(pretrained_model_name_or_path):
            resolved_model_card_file = pretrained_model_name_or_path
            is_local = True
        else:
            try:
                # Load from URL or cache if already cached
                resolved_model_card_file = cached_file(
                    pretrained_model_name_or_path,
                    filename=MODEL_CARD_NAME,
                    cache_dir=cache_dir,
                    proxies=proxies,
                    user_agent=user_agent,
                )
                if is_local:
                    logger.info(f"loading model card file {resolved_model_card_file}")
                else:
                    logger.info(f"loading model card file {MODEL_CARD_NAME} from cache at {resolved_model_card_file}")
                # Load model card
                modelcard = cls.from_json_file(resolved_model_card_file)

            except (EnvironmentError, json.JSONDecodeError):
                # We fall back on creating an empty model card
                modelcard = cls()

        # Update model card with kwargs if needed
        to_remove = []
        for key, value in kwargs.items():
            if hasattr(modelcard, key):
                setattr(modelcard, key, value)
                to_remove.append(key)
        for key in to_remove:
            kwargs.pop(key, None)

        logger.info(f"Model card: {modelcard}")
        if return_unused_kwargs:
            return modelcard, kwargs
        else:
            return modelcard

    @classmethod
    def from_dict(cls, json_object):
        """Constructs a `ModelCard` from a Python dictionary of parameters."""
        return cls(**json_object)

    @classmethod
    def from_json_file(cls, json_file):
        """Constructs a `ModelCard` from a json file of parameters."""
        with open(json_file, "r", encoding="utf-8") as reader:
            text = reader.read()
        dict_obj = json.loads(text)
        return cls(**dict_obj)

    def __eq__(self, other):
        return self.__dict__ == other.__dict__

    def __repr__(self):
        return str(self.to_json_string())

    def to_dict(self):
        """Serializes this instance to a Python dictionary."""
        output = copy.deepcopy(self.__dict__)
        return output

    def to_json_string(self):
        """Serializes this instance to a JSON string."""
        return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"

    def to_json_file(self, json_file_path):
        """Save this instance to a json file."""
        with open(json_file_path, "w", encoding="utf-8") as writer:


SUMMARY:
**ModelCard Class Summary**
==========================

**Class Purpose and Functionality**

The `ModelCard` class stores model card data and provides methods for loading, saving, and updating the model card. It is structured around the concept of model cards as described in the paper "Model Cards for Model Reporting" by Margaret Mitchell et al.

**Key Implementation Details**

* The class initializes with attributes such as `model_details`, `intended_use`, and `metrics`.
* The `save_pretrained` method saves the model card object to a directory or file.
* The `from_pretrained` class method instantiates a model card from a pre-trained model name or path.

**Class Method Signatures and Interfaces**

* `__init__`: Initializes the model card object with attributes
* `to_json_file`: Serializes the instance to a JSON string
* `from_dict`, `from_json_file`, and `from_pretrained`: Construct model cards from dictionaries, JSON files, or pre-trained models

**Notable Dependencies and Requirements**

* Python 3.x for compatibility
* `json` library for serializing data
* `cached_file` function from the `cached.io` library (not shown in the code snippet)

**Class Representation**

The class provides an instance of a model card as a dictionary of attributes, which can be serialized to a JSON string and saved to a file. It also offers methods for loading and updating model cards from pre-trained models.


key: chroma:document
string_value:
CODE:

def _listify(obj):
    if obj is None:
        return []
    elif isinstance(obj, str):
        return [obj]
    else:


SUMMARY:
**Summary**

The `_listify` function converts an input object into a list. It handles the following cases:

*   `None`: returns an empty list `[]`
*   `str`: wraps the string in a list `[obj]`
*   Other types: returns the object as is (e.g., integers, lists)

**Key Implementation Details**

*   Returns a specific type of data structure (`list`) based on input type
*   Uses `isinstance` to perform type checks

**Notable Dependencies or Requirements**

*   None


key: chroma:document
string_value:
CODE:

def _insert_values_as_list(metadata, name, values):
    if values is None:
        return metadata
    if isinstance(values, str):
        values = [values]
    values = [v for v in values if v is not None]
    if len(values) == 0:
        return metadata
    metadata[name] = values


SUMMARY:
**Function Summary: `_insert_values_as_list`**

Inserts values into the provided metadata and returns the updated metadata.

**Main Purpose**: Stores non-null values in the `metadata` dictionary under the specified `name`.

**Key Implementation Details**:

- Verifies input data integrity by filtering out `None` values.
- Enforces list-like structure for `values` input, converting it to a list if necessary.
- Adds values to the metadata using dictionary assignment.

**Notable Dependencies/Interfaces**:

* Uses `metadata` dictionary with exact key access (e.g., `metadata[name]`).
* Relying on conditional checks and iterated operations.


key: chroma:document
string_value:
CODE:

def infer_metric_tags_from_eval_results(eval_results):
    if eval_results is None:
        return {}
    result = {}
    for key in eval_results.keys():
        if key.lower().replace(" ", "_") in METRIC_TAGS:
            result[key.lower().replace(" ", "_")] = key
        elif key.lower() == "rouge1":
            result["rouge"] = key


SUMMARY:
**Infer Metric Tags from Evaluation Results**

*Purpose: Reconstructs metric tags from evaluation results for inference purposes.*

**Primary Purpose:**
Infer metric tags by matching keys in `eval_results` with known metric tags in `METRIC_TAGS`.

**Key Implementation Details:**

* Iterates through keys in `eval_results` and checks if any are matched against `METRIC_TAGS`.
* If a match is found, assigns the original key (before whitespace removal) to the result dictionary with modified key (after whitespace removal).
* Special case for "rouge1" key, which maps to "rouge".

**Notable Dependencies/Requirements:**

* `METRIC_TAGS`: a list of known metric tags (not shown in this code snippet).
* No external dependencies listed.

**Code Structure and Design Choices:**

* Simple iterative approach using Python dictionary iteration.
* Efficient use of modified keys for mapping results, reducing potential issues with key conflicts.


key: chroma:document
string_value:
CODE:

def _insert_value(metadata, name, value):
    if value is None:
        return metadata
    metadata[name] = value


SUMMARY:
**Summary:**

`_insert_value` function inserts a new key-value pair into `metadata` while handling the case when the value is `None`. Primary purpose: update a dictionary with a given name and value.

**Context:** Updating metadata with a specific key-value pair.

**Key details:**

* **Function signature:** `_insert_value(metadata, name, value)`
* **Implementation:** if `value` is `None`, returns original `metadata`; otherwise, updates `metadata[name] = value`
* **Inherited interfaces:** No explicit inheritance; relying on dictionary operations.


key: chroma:document
string_value:
CODE:

def is_hf_dataset(dataset):
    if not is_datasets_available():
        return False

    from datasets import Dataset, IterableDataset



SUMMARY:
**Summary**

This is a Python function `is_hf_dataset` that takes a `dataset` as input and returns a boolean indicating whether it's an HDF dataset.

### Key Details

* Checks if the necessary `datasets` library is available.
* Verifies if the provided `dataset` is either of type `Dataset` or `IterableDataset`.


key: chroma:document
string_value:
CODE:

def _get_mapping_values(mapping):
    result = []
    for v in mapping.values():
        if isinstance(v, (tuple, list)):
            result += list(v)
        else:
            result.append(v)


SUMMARY:
**Summary:**

`_get_mapping_values` function:

* Takes a dictionary `mapping` as input
* Flattens nested values (tuples or lists) into a single list
* Returns the flattened values in ascending order

**Key details:**

* Implemented using a simple for loop and type checks (`isinstance`)
* Uses the `+=` operator to prepend new elements to an existing list (`result += list(v)`)

**Notable dependencies/requirements:**

* None mentioned, but likely relies on built-in Python data structures and types.


key: chroma:document
string_value:
CODE:

def parse_keras_history(logs):
    """
    Parse the `logs` of either a `keras.History` object returned by `model.fit()` or an accumulated logs `dict`
    passed to the `PushToHubCallback`. Returns lines and logs compatible with those returned by `parse_log_history`.
    """
    if hasattr(logs, "history"):
        # This looks like a `History` object
        if not hasattr(logs, "epoch"):
            # This history looks empty, return empty results
            return None, [], {}
        logs.history["epoch"] = logs.epoch
        logs = logs.history
    else:
        # Training logs is a list of dicts, let's invert it to a dict of lists to match a History object
        logs = {log_key: [single_dict[log_key] for single_dict in logs] for log_key in logs[0]}

    lines = []
    for i in range(len(logs["epoch"])):
        epoch_dict = {log_key: log_value_list[i] for log_key, log_value_list in logs.items()}
        values = {}
        for k, v in epoch_dict.items():
            if k.startswith("val_"):
                k = "validation_" + k[4:]
            elif k != "epoch":
                k = "train_" + k
            splits = k.split("_")
            name = " ".join([part.capitalize() for part in splits])
            values[name] = v
        lines.append(values)

    eval_results = lines[-1]



SUMMARY:
**Summary:**

### Function Purpose and Functionality

`parse_keras_history(logs)` parses the Keras training history object or an accumulated log dictionary into a structured format.

**Main Purpose:** Convert Keras history data to match `parse_log_history` output, allowing for easier analysis and comparison of model performance metrics across different epochs.

### Key Implementation Details

*   Handles both instances of `keras.History` objects and accumulated log dictionaries as input.
*   Inverts the log dictionary into a format similar to `keras.History` if it's not already in that form.
*   Iterates over each epoch, normalizing metric keys (e.g., "val_loss" becomes "validation_loss") and extracting relevant values.

**Notable Dependencies**

*   Keras library
*   ModelFit()
*   PushToHubCallback

### Notable Code Sections

*   Conditionals for handling `History` objects vs. log dictionaries:
    ```python
if hasattr(logs, "history"):
    # Handle History object
else:
    # Invert log dictionary to match History format
```


key: chroma:document
string_value:
CODE:

def parse_log_history(log_history):
    """
    Parse the `log_history` of a Trainer to get the intermediate and final evaluation results.
    """
    idx = 0
    while idx < len(log_history) and "train_runtime" not in log_history[idx]:
        idx += 1

    # If there are no training logs
    if idx == len(log_history):
        idx -= 1
        while idx >= 0 and "eval_loss" not in log_history[idx]:
            idx -= 1

        if idx >= 0:
            return None, None, log_history[idx]
        else:
            return None, None, None

    # From now one we can assume we have training logs:
    train_log = log_history[idx]
    lines = []
    training_loss = "No log"
    for i in range(idx):
        if "loss" in log_history[i]:
            training_loss = log_history[i]["loss"]
        if "eval_loss" in log_history[i]:
            metrics = log_history[i].copy()
            _ = metrics.pop("total_flos", None)
            epoch = metrics.pop("epoch", None)
            step = metrics.pop("step", None)
            _ = metrics.pop("eval_runtime", None)
            _ = metrics.pop("eval_samples_per_second", None)
            _ = metrics.pop("eval_steps_per_second", None)
            _ = metrics.pop("eval_jit_compilation_time", None)
            values = {"Training Loss": training_loss, "Epoch": epoch, "Step": step}
            for k, v in metrics.items():
                if k == "eval_loss":
                    values["Validation Loss"] = v
                else:
                    splits = k.split("_")
                    name = " ".join([part.capitalize() for part in splits[1:]])
                    values[name] = v
            lines.append(values)

    idx = len(log_history) - 1
    while idx >= 0 and "eval_loss" not in log_history[idx]:
        idx -= 1

    if idx > 0:
        eval_results = {}
        for key, value in log_history[idx].items():
            if key.startswith("eval_"):
                key = key[5:]
            if key not in ["runtime", "samples_per_second", "steps_per_second", "epoch", "step"]:
                camel_cased_key = " ".join([part.capitalize() for part in key.split("_")])
                eval_results[camel_cased_key] = value
        return train_log, lines, eval_results
    else:


SUMMARY:
**Summary**

`parse_log_history` function:

* Parses log history to extract intermediate and final evaluation results for a `Trainer`.
* Returns tuple containing training log, epoch-wise metrics lines, and final evaluation metrics.

**Key Implementation Details**

* Iterates over log history to find first occurrence of "train_runtime" and then extracts relevant lines.
* Extracts training loss from earlier logs and populates metrics dictionary with required key-value pairs.
* Creates list of epoch-wise metrics lines by iterating over intermediate logs and appending values.
* Iterates over final logs to extract evaluation metrics (excluding "runtime", "samples_per_second", etc.).
* Returns training log, epoch-wise metrics lines, and final evaluation metrics.

**Notable Dependencies or Requirements**

* `log_history`: Input log history string or list of strings to parse.
* Uses:
	+ `metrics` dictionary to store key-value pairs for epoch-wise metrics lines.
	+ `_` (implicit assignment) to remove unnecessary values from dictionaries.


key: chroma:document
string_value:
CODE:

def extract_hyperparameters_from_keras(model):
    from .modeling_tf_utils import keras

    hyperparameters = {}
    if hasattr(model, "optimizer") and model.optimizer is not None:
        hyperparameters["optimizer"] = model.optimizer.get_config()
    else:
        hyperparameters["optimizer"] = None
    hyperparameters["training_precision"] = keras.mixed_precision.global_policy().name



SUMMARY:
**Function Summary: `extract_hyperparameters_from_keras`**

Extracts hyperparameter values from a Keras model.

**Key Details:**

*   Retrieves optimizer settings, including configuration and global precision policy.
*   Utilizes `keras.mixed_precision.global_policy().name` to determine training precision setting.

**Method Signatures and Interfaces:**

*   Returns a dictionary (`hyperparameters`) containing extracted hyperparameter values.

**Dependent Functions/Classes:**

*   `.modeling_tf_utils.keras`
*   Keras model's optimizer (conditionally)

**External Dependencies:**

*   Keras library (imported as `keras` from `.modeling_tf_utils`)
*   TensorFlow Mixed Precision Policy library


key: chroma:document
string_value:
CODE:

def _maybe_round(v, decimals=4):
    if isinstance(v, float) and len(str(v).split(".")) > 1 and len(str(v).split(".")[1]) > decimals:
        return f"{v:.{decimals}f}"


SUMMARY:
**Summary:**

`_maybe_round` function formats a float value to specify decimal places, rounding if necessary.

* Purpose:
	+ Rounds or truncates floating-point numbers within the specified decimal place range.
	+ Defaults to 4 decimal places.
* Implementation Details:
	+ Uses conditional statement with `isinstance()`, string manipulation (`str(v).split(".")`), and f-string formatting (`f"{v:.{decimals}f}"`).
* Notable Dependencies/Requirements: None.

**Key Method Signatures and Interfaces:** 
- `formatting(f_string, name, num_decimal_places)`

**External Dependencies:** N/A

**Inherited or Implemented Interfaces:** None.


key: chroma:document
string_value:
CODE:

def _regular_table_line(values, col_widths):
    values_with_space = [f"| {v}" + " " * (w - len(v) + 1) for v, w in zip(values, col_widths)]


SUMMARY:
**Summary:**

`_regular_table_line` function formats table row data with aligned columns.

**Key Details:**

* Takes two parameters: `values` (table content) and `col_widths` (column widths)
* Uses string formatting to add whitespace and create aligned columns
* Joins formatted values with a newline character

**Implementation Notable:**

* Utilizes the `"%"` alignment format specifer to align columns

**Dependencies/Requirements:**

* None, but assumes use of Python's built-in `zip`, `f-string`, and string formatting functionality.


key: chroma:document
string_value:
CODE:

def _second_table_line(col_widths):
    values = ["|:" + "-" * w + ":" for w in col_widths]


SUMMARY:
**Summary:**

`_second_table_line` function:

* Takes `col_widths` as input, where each width represents a table column
* Generates a string representation of the line separating two tables by iterating over `col_widths`, concatenating colon-separated dashes with pipes (`|:`)
* Returns the generated separator string along with a newline character (`\n`) at the end

**Key Implementation Details:**

* Primary function signature: `def _second_table_line(col_widths) -> str`
* Utilizes list comprehension to generate the separator string efficiently
* Leverages the `join` method to concatenate the values, eliminating unnecessary concatenation operations
* Includes a newline character (`\n`) at the end of the returned string


key: chroma:document
string_value:
CODE:

def make_markdown_table(lines):
    """
    Create a nice Markdown table from the results in `lines`.
    """
    if lines is None or len(lines) == 0:
        return ""
    col_widths = {key: len(str(key)) for key in lines[0].keys()}
    for line in lines:
        for key, value in line.items():
            if col_widths[key] < len(_maybe_round(value)):
                col_widths[key] = len(_maybe_round(value))

    table = _regular_table_line(list(lines[0].keys()), list(col_widths.values()))
    table += _second_table_line(list(col_widths.values()))
    for line in lines:
        table += _regular_table_line([_maybe_round(v) for v in line.values()], list(col_widths.values()))


SUMMARY:
**Concise Summary**

The `make_markdown_table` function creates a Markdown table from the provided input data, represented as a dictionary of lines. It calculates column widths based on row values and generates a complete table, including header and separator lines.

**Details:**

* **Functionality:** Calculates column widths for each key in the input data and constructs a Markdown table.
* **Key Algorithm/Pattern:** Uses dictionaries to store and manipulate data, with iterative processing of each line item.

**Important Method Signatures and Interfaces:**

* `_regular_table_line()`: Takes lists of headers and values as arguments
* `_second_table_line()`: Takes list of column widths as an argument
* `_maybe_round()`: Returns rounded value ( exact implementation not shown)

**Notable Dependencies or Requirements:**

* External dependency: None identified
* Inherited/Implemented interfaces: Assuming existence of `_regular_table_line`, `_second_table_line` and `_maybe_round` functions/functions with defined interfaces.


key: chroma:document
string_value:
CODE:

def extract_hyperparameters_from_trainer(trainer):
    hyperparameters = {k: getattr(trainer.args, k) for k in _TRAINING_ARGS_KEYS}

    if trainer.args.parallel_mode not in [ParallelMode.NOT_PARALLEL, ParallelMode.NOT_DISTRIBUTED]:
        hyperparameters["distributed_type"] = (
            "multi-GPU" if trainer.args.parallel_mode == ParallelMode.DISTRIBUTED else trainer.args.parallel_mode.value
        )
    if trainer.args.world_size > 1:
        hyperparameters["num_devices"] = trainer.args.world_size
    if trainer.args.gradient_accumulation_steps > 1:
        hyperparameters["gradient_accumulation_steps"] = trainer.args.gradient_accumulation_steps

    total_train_batch_size = (
        trainer.args.train_batch_size * trainer.args.world_size * trainer.args.gradient_accumulation_steps
    )
    if total_train_batch_size != hyperparameters["train_batch_size"]:
        hyperparameters["total_train_batch_size"] = total_train_batch_size
    total_eval_batch_size = trainer.args.eval_batch_size * trainer.args.world_size
    if total_eval_batch_size != hyperparameters["eval_batch_size"]:
        hyperparameters["total_eval_batch_size"] = total_eval_batch_size

    if trainer.args.optim:
        optimizer_name = trainer.args.optim
        optimizer_args = trainer.args.optim_args if trainer.args.optim_args else "No additional optimizer arguments"

        if "adam" in optimizer_name.lower():
            hyperparameters["optimizer"] = (
                f"Use {optimizer_name} with betas=({trainer.args.adam_beta1},{trainer.args.adam_beta2}) and"
                f" epsilon={trainer.args.adam_epsilon} and optimizer_args={optimizer_args}"
            )
        else:
            hyperparameters["optimizer"] = f"Use {optimizer_name} and the args are:\n{optimizer_args}"

    hyperparameters["lr_scheduler_type"] = trainer.args.lr_scheduler_type.value
    if trainer.args.warmup_ratio != 0.0:
        hyperparameters["lr_scheduler_warmup_ratio"] = trainer.args.warmup_ratio
    if trainer.args.warmup_steps != 0.0:
        hyperparameters["lr_scheduler_warmup_steps"] = trainer.args.warmup_steps
    if trainer.args.max_steps != -1:
        hyperparameters["training_steps"] = trainer.args.max_steps
    else:
        hyperparameters["num_epochs"] = trainer.args.num_train_epochs

    if trainer.args.fp16:
        if trainer.use_apex:
            hyperparameters["mixed_precision_training"] = f"Apex, opt level {trainer.args.fp16_opt_level}"
        else:
            hyperparameters["mixed_precision_training"] = "Native AMP"

    if trainer.args.label_smoothing_factor != 0.0:
        hyperparameters["label_smoothing_factor"] = trainer.args.label_smoothing_factor



SUMMARY:
**Function Summary:**

`extract_hyperparameters_from_trainer`

**Purpose and Functionality:**

This function extracts key hyperparameters from a `Trainer` object, including distributed training settings, optimizer configurations, learning rate schedulers, and mixed precision training options.

**Key Implementation Details:**

* Extracts hyperparameter values using `getattr` for each key in `_TRAINING_ARGS_KEYS`
* Computes total train and eval batch sizes based on `world_size`, `gradient_accumulation_steps`, and other training settings
* Processes optimizer configurations, including AdamW and mixed precision training options
* Updates hyperparameters with learning rate scheduler settings and label smoothing factor

**Notable Dependencies or Requirements:**

* `Trainer` object with attributes for training settings (e.g., `_TRAINING_ARGS_KEYS`, `parallel_mode`, `world_size`)
* `_TRAINING_ARGS_KEYS` list defining valid hyperparameter keys
* `ParallelMode` enum class for parallel mode determination

**Interfaces:**

* Uses ` getattr` to extract attribute values from the `Trainer` object's namespace.

This summary preserves exact class/function variable names and provides a concise overview of the function's purpose, key implementation details, and dependencies or requirements.


key: chroma:document
string_value:
CODE:

class ChannelDimension(ExplicitEnum):
    FIRST = "channels_first"


SUMMARY:
**Class Summary**

`ChannelDimension` is an `ExplicitEnum` that defines two possible dimension configurations for channels.

**Key Implementation Details**
- `FIRST`: Channels are presented in the order of their most recently added dimensions at the beginning (`channels_first`).
- `LAST`: Channels are presented in the reverse order of their most recently added dimensions at the end (`channels_last`).

**Notable Dependencies or Requirements**

- Inherits from `ExplicitEnum`, a base class in Python standard library.

No external dependencies mentioned.


key: chroma:document
string_value:
CODE:

class AnnotationFormat(ExplicitEnum):
    COCO_DETECTION = "coco_detection"


SUMMARY:
**Summary**

`AnnotationFormat` is an `ExplicitEnum` class defining two annotation formats: `COCO_DETECTION` and `COCO_PANOPTIC`. These values are used to specify the format of annotations in object detection models, such as COCO (Common Objects in Context) detection. No external dependencies or notable implementation details are present.


key: chroma:document
string_value:
CODE:

class AnnotionFormat(ExplicitEnum):
    COCO_DETECTION = AnnotationFormat.COCO_DETECTION.value


SUMMARY:
**Summary:**

This code defines an enumeration `AnnotionFormat` with two values, `COCO_DETECTION` and `COCO_PANOPTIC`, which mirror the corresponding constants in the `AnnotationFormat` class.

**Key Details:**

* **Implementation:** The values are assigned using the `.value` attribute of the parent `AnnotationFormat` enum.
* **Primary Purpose:** To provide a readable and maintainable way to represent different annotation formats.
* **Notable Interfaces:** Inherited from `ExplicitEnum`, which typically provides additional utility methods for enum-like behavior.

**Dependences:**

* `AnnotationFormat` class (parent enum)


key: chroma:document
string_value:
CODE:

def is_pil_image(img):


SUMMARY:
**Summary:**

`is_pil_image` function checks if an image object `img` is a valid Pillow Image object. It returns `True` if both `vision` functionality is available and the input `img` is an instance of PIL's `Image.Image`.

**Key Implementation Details:**

- Checks availability of `vision` functionality using `is_vision_available()` 
- Verifies image type using `isinstance(img, PIL.Image.Image)`

**External Dependencies:**

- Requires Pillow library (`PIL`)


key: chroma:document
string_value:
CODE:

class ImageType(ExplicitEnum):
    PIL = "pillow"
    TORCH = "torch"
    NUMPY = "numpy"
    TENSORFLOW = "tensorflow"


SUMMARY:
**Summary:**

The `ImageType` class ( ExplicitEnum ) defines a set of enumeration values representing different image processing technologies:

*   `PIL`, mapped to "pillow"
*   `TORCH`, mapped to "torch"
*   `NUMPY`, mapped to "numpy"
*   `TENSORFLOW`, mapped to "tensorflow"
*   `JAX`, mapped to "jax"

This class likely serves as a way to restrict or formalize image processing technology choices within an application or library.


key: chroma:document
string_value:
CODE:

def get_image_type(image):
    if is_pil_image(image):
        return ImageType.PIL
    if is_torch_tensor(image):
        return ImageType.TORCH
    if is_numpy_array(image):
        return ImageType.NUMPY
    if is_tf_tensor(image):
        return ImageType.TENSORFLOW
    if is_jax_tensor(image):
        return ImageType.JAX


SUMMARY:
**Function Summary:**

`get_image_type(image)` 
=========================

Determines the type of an input image based on its underlying format.

*   Purpose:
    *   Identifies the image type, specifically its technology stack (PIL, Torch, NumPy, TensorFlow, JAX).
    *   Raises a ValueError if the image type is unrecognized.

**Key Implementation Details:**

*   Uses multiple conditional checks to determine the image type.
*   Utilizes external functions:
    *   `is_pil_image(image)`: Checks if the image is in the Python Imaging Library (PIL) format.
    *   `is_torch_tensor(image)`, `is_numpy_array(image)`, `is_tf_tensor(image)`, and `is_jax_tensor(image)`: Check for respective formats in Torch, NumPy, TensorFlow, and JAX.

**Notable Dependencies:**

*   External dependencies include PIL, Torch (Torch library), NumPy, TensorFlow, and JAX.

Note that the function's exact type (i.e., Python function or class method) is not specified due to an lack of context; however, it is likely a utility function in a larger framework such as image processing.


key: chroma:document
string_value:
CODE:

def is_valid_image(img):


SUMMARY:
**Summary:**

```python
def is_valid_image(img):
```

*   **Purpose:** Determines if an input image is valid by checking its type against multiple formats using separate functions (`is_pil_image`, `is_numpy_array`, `is_torch_tensor`, `is_tf_tensor`, and `is_jax_tensor`).
*   **Key Implementation Details:**
    *   Uses multiple checks for different image formats.
    *   Returns a boolean value indicating validation status.

**Related Imports/Classes/Functions:**

No explicit references, but it's assumed that the functions (`is_pil_image`, `is_numpy_array`, `is_torch_tensor`, `is_tf_tensor`, and `is_jax_tensor`) are defined elsewhere in the codebase.
```python
import PIL
import numpy as np
import torch
import tensorflow as tf
import jax
```
Note: The exact import statements might be different depending on the actual implementation.


key: chroma:document
string_value:
CODE:

def valid_images(imgs):
    # If we have an list of images, make sure every image is valid
    if isinstance(imgs, (list, tuple)):
        for img in imgs:
            if not valid_images(img):
                return False
    # If not a list of tuple, we have been given a single image or batched tensor of images
    elif not is_valid_image(imgs):
        return False


SUMMARY:
**Summary:**

The `valid_images` function checks whether one or more images (or batches of images) are valid. It recursively verifies each image in the input list, returning `False` if any image is invalid. The primary purpose is to validate image integrity.

**Key Analysis Points:**

* Algorithm: Recursive image validation using list/tuple traversal
* Design Choice: Separate recursive function for handling nested lists/ranges
* Important Method Signatures:
	+ `valid_images(img)` (base case)
	+ `is_valid_image(imgs)` (used to verify single/batch images)

**Notable Dependencies:**

* External dependency on an implicit 'image validation' function (`is_valid_image`)
* Inherent assumptions about the image data structure and its validity checks


key: chroma:document
string_value:
CODE:

def is_batched(img):
    if isinstance(img, (list, tuple)):
        return is_valid_image(img[0])


SUMMARY:
**Function Summary: `is_batched`**

The `is_batched` function checks if an image input (`img`) is batched. It performs the following operations:

* If the input is a list or tuple, it calls `is_valid_image` on its first element (`img[0]`).
* Otherwise, it returns `False`.

**Key Implementation Details:**

* Uses `isinstance` to check the type of the input.
* Calls another function `is_valid_image` with the image data.

**Notable Dependencies/Requirements:**

* `is_valid_image` function (not shown in this code snippet).

This summary provides a concise overview of the function's purpose and key implementation details, without including unnecessary information.


key: chroma:document
string_value:
CODE:

def is_scaled_image(image: np.ndarray) -> bool:
    """
    Checks to see whether the pixel values have already been rescaled to [0, 1].
    """
    if image.dtype == np.uint8:
        return False

    # It's possible the image has pixel values in [0, 255] but is of floating type


SUMMARY:
**Summary**

```markdown
# Function: is_scaled_image
## Purpose
Checks if an image has pixel values in the [0, 1] range.

### Description

Returns `True` if the pixel values are already rescaled to `[0, 1]`. Checks for the following conditions:
- The image uses **unsigned 8-bit integer** data type (`np.uint8`), which implies scaling is not applied.
- The minimum and maximum pixel values are within the scaled range `[0, 1]`.

### Implementation

```python
def is_scaled_image(image: np.ndarray) -> bool:
    # Checks if image uses unsigned 8-bit integer data type and pixel values are in [0, 255]
    # Assume floating-point representation is present if min and max are outside [0, 1] range.
```

**Key Insights**

- The function assumes a `np.uint8` data type unless the pixel values indicate another representation.
- It relies on `np.min(image)` and `np.max(image)` being greater than or equal to 0 and less than or equal to 1 for floating-point images.


key: chroma:document
string_value:
CODE:

def make_list_of_images(images, expected_ndims: int = 3) -> List[ImageInput]:
    """
    Ensure that the input is a list of images. If the input is a single image, it is converted to a list of length 1.
    If the input is a batch of images, it is converted to a list of images.

    Args:
        images (`ImageInput`):
            Image of images to turn into a list of images.
        expected_ndims (`int`, *optional*, defaults to 3):
            Expected number of dimensions for a single input image. If the input image has a different number of
            dimensions, an error is raised.
    """
    if is_batched(images):
        return images

    # Either the input is a single image, in which case we create a list of length 1
    if isinstance(images, PIL.Image.Image):
        # PIL images are never batched
        return [images]

    if is_valid_image(images):
        if images.ndim == expected_ndims + 1:
            # Batch of images
            images = list(images)
        elif images.ndim == expected_ndims:
            # Single image
            images = [images]
        else:
            raise ValueError(
                f"Invalid image shape. Expected either {expected_ndims + 1} or {expected_ndims} dimensions, but got"
                f" {images.ndim} dimensions."
            )
        return images
    raise ValueError(
        "Invalid image type. Expected either PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or "
        f"jax.ndarray, but got {type(images)}."


SUMMARY:
**Function Summary: `make_list_of_images`**

Converts input images to a list of images while ensuring the expected number of dimensions.

**Purpose and Functionality**
This function takes an image or batch of images as input, checks its type and dimensionality, and returns a list of images. If the input is a single image, it is converted to a list with one element. If the input is already a list of images, no conversion is performed.

**Key Implementation Details**

* Uses `PIL Image` class for single-image inputs
* Verifies expected number of dimensions (3 by default) using `ndim` attribute
* Converts batch of images or single image to list using indexing or slicing

**Notable Dependencies and Requirements**
- PIL (Python Imaging Library)
- Numpy
- PyTorch/TensorFlow/JuliaNN (for valid image types)

Please note that exact method signatures, interfaces, and class names are preserved in the summary for accuracy.


key: chroma:document
string_value:
CODE:

def to_numpy_array(img) -> np.ndarray:
    if not is_valid_image(img):
        raise ValueError(f"Invalid image type: {type(img)}")

    if is_vision_available() and isinstance(img, PIL.Image.Image):
        return np.array(img)


SUMMARY:
**Summary**

A function `tonumpy_array` that converts an input image to a NumPy array:
- Validates the image type using `is_valid_image`.
  * Relying on explicit comparison `type(img)`.
  - Checks if vision is available and if the image is of type PIL.Image.Image.
    * Import of PIL library seems implied, assume `PIL` in scope.
  - If valid and compatible, returns the array conversion using NumPy's `array`.
  - Otherwise, falls back to `to_numpy` method.

**Key Details**

* Method return type: `np.ndarray`
* Interface checked for `vision` availability
* Image validation: `is_valid_image`

**Notable Dependency**
* `PIL.Image.Image`: implied import


key: chroma:document
string_value:
CODE:

def infer_channel_dimension_format(
    image: np.ndarray, num_channels: Optional[Union[int, Tuple[int, ...]]] = None
) -> ChannelDimension:
    """
    Infers the channel dimension format of `image`.

    Args:
        image (`np.ndarray`):
            The image to infer the channel dimension of.
        num_channels (`int` or `Tuple[int, ...]`, *optional*, defaults to `(1, 3)`):
            The number of channels of the image.

    Returns:
        The channel dimension of the image.
    """
    num_channels = num_channels if num_channels is not None else (1, 3)
    num_channels = (num_channels,) if isinstance(num_channels, int) else num_channels

    if image.ndim == 3:
        first_dim, last_dim = 0, 2
    elif image.ndim == 4:
        first_dim, last_dim = 1, 3
    else:
        raise ValueError(f"Unsupported number of image dimensions: {image.ndim}")

    if image.shape[first_dim] in num_channels and image.shape[last_dim] in num_channels:
        logger.warning(
            f"The channel dimension is ambiguous. Got image shape {image.shape}. Assuming channels are the first dimension."
        )
        return ChannelDimension.FIRST
    elif image.shape[first_dim] in num_channels:
        return ChannelDimension.FIRST
    elif image.shape[last_dim] in num_channels:
        return ChannelDimension.LAST


SUMMARY:
**Summary**

This function, `infer_channel_dimension_format`, determines the channel dimension format of an input image. The main purpose is to classify the channel dimension as either `.FIRST` or `.LAST`.

**Key Implementation Details**

* Checks if the image has 3 or 4 dimensions and adjusts the dimension indices accordingly.
* Verifies if the first/last dimension's size is present in `num_channels`. Falls back to default values if ambiguous.

**Notable Dependencies/Requirements**

* None

**Returns:**

* `ChannelDimension.FIRST` or `ChannelDimension.LAST`, indicating whether the channel dimension is in the first or last position, respectively.


key: chroma:document
string_value:
CODE:

def get_channel_dimension_axis(
    image: np.ndarray, input_data_format: Optional[Union[ChannelDimension, str]] = None
) -> int:
    """
    Returns the channel dimension axis of the image.

    Args:
        image (`np.ndarray`):
            The image to get the channel dimension axis of.
        input_data_format (`ChannelDimension` or `str`, *optional*):
            The channel dimension format of the image. If `None`, will infer the channel dimension from the image.

    Returns:
        The channel dimension axis of the image.
    """
    if input_data_format is None:
        input_data_format = infer_channel_dimension_format(image)
    if input_data_format == ChannelDimension.FIRST:
        return image.ndim - 3
    elif input_data_format == ChannelDimension.LAST:
        return image.ndim - 1


SUMMARY:
**Function Summary**

`get_channel_dimension_axis(image, input_data_format=None) -> int`

Retrieves the channel dimension axis of an image based on its `input_data_format`.

**Key Details**

* Returns `image.ndim - 3` if `input_data_format` is `ChannelDimension.FIRST`
* Returns `image(ndim - 1)` if `input_data_format` is `ChannelDimension.LAST`

**Dependencies/Requirements**

* `infer_channel_dimension_format(image)`
* `ChannelDimension`
* Supports channels in dimensions: `(2, 128`, ...,), (3, 256, ...)

**Notes on Implementation**

Uses a simple conditional statement to handle supported data formats. Raises an error for unsupported formats.


key: chroma:document
string_value:
CODE:

def get_image_size(image: np.ndarray, channel_dim: ChannelDimension = None) -> Tuple[int, int]:
    """
    Returns the (height, width) dimensions of the image.

    Args:
        image (`np.ndarray`):
            The image to get the dimensions of.
        channel_dim (`ChannelDimension`, *optional*):
            Which dimension the channel dimension is in. If `None`, will infer the channel dimension from the image.

    Returns:
        A tuple of the image's height and width.
    """
    if channel_dim is None:
        channel_dim = infer_channel_dimension_format(image)

    if channel_dim == ChannelDimension.FIRST:
        return image.shape[-2], image.shape[-1]
    elif channel_dim == ChannelDimension.LAST:
        return image.shape[-3], image.shape[-2]
    else:


SUMMARY:
**Summary**

`get_image_size` function returns the dimensions of an input image. It takes two parameters:

*   `image`: A 3D numpy array representing the image.
*   `channel_dim`: The dimension where the channel dimension is located (optional; defaults to inference).

The function returns a tuple containing the height and width of the image.

**Implementation Details**

*   Inferences channel dimension if provided.
*   Returns dimensions using `image.shape[-2]` and `image.shape[-1]` for `ChannelDimension.FIRST` and `ChannelDimension.LAST`, respectively.

**Dependencies/Requirements**

*   `infer_channel_dimension_format`: A function to infer the channel dimension format from an image (not shown in this code snippet).
*   `ChannelDimension`: An enum representing the possible data formats (FIRST and LAST).

**Method Signatures and Interfaces**

*   `image.shape[-2]` and `image.shape[-1]`.

Note that the actual implementation of `infer_channel_dimension_format` is not provided, as it is assumed to be a separate function. Additionally, the `ChannelDimension` enum is also not defined in this code snippet.


key: chroma:document
string_value:
CODE:

def is_valid_annotation_coco_detection(annotation: Dict[str, Union[List, Tuple]]) -> bool:
    if (
        isinstance(annotation, dict)
        and "image_id" in annotation
        and "annotations" in annotation
        and isinstance(annotation["annotations"], (list, tuple))
        and (
            # an image can have no annotations
            len(annotation["annotations"]) == 0 or isinstance(annotation["annotations"][0], dict)
        )
    ):
        return True


SUMMARY:
**Summary: `is_valid_annotation_coco_detection` Function**

This function determines whether a given annotation is valid for COCO Detection format. It checks if the input `annotation` adheres to the following structure:
- Has an "image_id" key
- Has an "annotations" key containing a list or tuple of annotations, with at least one being a dictionary

**Implementation Details:**

* Uses `isinstance()` to check type and validate nested structures.
* Returns `True` if the annotation is valid and falsy otherwise.

**Notable Dependencies:**

* None explicitly listed
* May rely on the `collections` library ( implicit: `Dict`, `Union`, `List`, `Tuple`)


key: chroma:document
string_value:
CODE:

def is_valid_annotation_coco_panoptic(annotation: Dict[str, Union[List, Tuple]]) -> bool:
    if (
        isinstance(annotation, dict)
        and "image_id" in annotation
        and "segments_info" in annotation
        and "file_name" in annotation
        and isinstance(annotation["segments_info"], (list, tuple))
        and (
            # an image can have no segments
            len(annotation["segments_info"]) == 0 or isinstance(annotation["segments_info"][0], dict)
        )
    ):
        return True


SUMMARY:
**Summary: `is_valid_annotation_coco_panoptic` Function**

This function checks if an annotation dictionary is valid for the COCO panoptic segmentation format. It verifies that the input annotation contains required keys ("image_id", "segments_info", and "file_name") and ensures that "segments_info" is either a list or tuple containing segments with correct structure.

**Key implementation details:**

* Uses `isinstance` checks to verify types of annotations
* Enforces strict structure for valid "segments_info"
* Returns `True` if annotation is valid, `False` otherwise

**Notable dependencies and requirements:**

* `Dict[str, Union[List, Tuple]]`: Input annotation type hint
* `Len(annotation["segments_info"]) == 0 or isinstance(annotation["segments_info"][0], dict)`: Condition for valid segments format
* No external dependencies listed


key: chroma:document
string_value:
CODE:

def valid_coco_detection_annotations(annotations: Iterable[Dict[str, Union[List, Tuple]]]) -> bool:


SUMMARY:
**Summary:**

```python
def valid_coco_detection_annotations(annotations: Iterable[Dict[str, Union[List, Tuple]]]) -> bool:
    # Returns True if all COCO detection annotations are valid, False otherwise

    def is_valid_annotation_coco_detection(ann: Dict[str, Union[List, Tuple]]) -> bool:
        # Internal function to validate a single annotation
        pass  # implementation not shown in this snippet
```

**Key Analysis Points:**

- **Primary Purpose:** Checks if all COCO detection annotations are valid.
- **Implementation Focus:** The function iterates over the annotated objects using an iterable and calls `is_valid_annotation_coco_detection` on each one, returning a boolean result based on all values being True.


key: chroma:document
string_value:
CODE:

def valid_coco_panoptic_annotations(annotations: Iterable[Dict[str, Union[List, Tuple]]]) -> bool:


SUMMARY:
**Summary**

This function, `valid_coco_panoptic_annotations`, checks the validity of COCO Panoptic annotations. It ensures that all annotations in the input iterable conform to the annotation format using the `is_valid_annotation_coco_panoptic` helper function. The function returns `True` if all annotations are valid.

**Key Implementation Details**

* The function uses the `all` function with a generator expression to iterate over annotated objects (`ann`) in the input iterable.
* It relies on the `is_valid_annotation_coco_panoptic` function, which is not shown in this code snippet, to determine whether each annotation is valid.
* The function returns the result of the `all` iteration.

**Notable Dependencies or Requirements**

* A function or class named `is_valid_annotation_coco_panoptic`, whose implementation is not provided.


key: chroma:document
string_value:
CODE:

def load_image(image: Union[str, "PIL.Image.Image"], timeout: Optional[float] = None) -> "PIL.Image.Image":
    """
    Loads `image` to a PIL Image.

    Args:
        image (`str` or `PIL.Image.Image`):
            The image to convert to the PIL Image format.
        timeout (`float`, *optional*):
            The timeout value in seconds for the URL request.

    Returns:
        `PIL.Image.Image`: A PIL Image.
    """
    requires_backends(load_image, ["vision"])
    if isinstance(image, str):
        if image.startswith("http://") or image.startswith("https://"):
            # We need to actually check for a real protocol, otherwise it's impossible to use a local file
            # like http_huggingface_co.png
            image = PIL.Image.open(BytesIO(requests.get(image, timeout=timeout).content))
        elif os.path.isfile(image):
            image = PIL.Image.open(image)
        else:
            if image.startswith("data:image/"):
                image = image.split(",")[1]

            # Try to load as base64
            try:
                b64 = base64.decodebytes(image.encode())
                image = PIL.Image.open(BytesIO(b64))
            except Exception as e:
                raise ValueError(
                    f"Incorrect image source. Must be a valid URL starting with `http://` or `https://`, a valid path to an image file, or a base64 encoded string. Got {image}. Failed with {e}"
                )
    elif isinstance(image, PIL.Image.Image):
        image = image
    else:
        raise TypeError(
            "Incorrect format used for image. Should be an url linking to an image, a base64 string, a local path, or a PIL image."
        )
    image = PIL.ImageOps.exif_transpose(image)
    image = image.convert("RGB")


SUMMARY:
**Summary:**

### Function Purpose and Functionality
`load_image`: A function to load an image to a PIL Image format, supporting various input formats - string (URL, base64), path, or PIL Image. Loads the requested image with a given timeout.

### Key Implementation Details
- **Algorithm:** Converts input image from various sources to PIL Image, applying necessary operations such as URL request, decoding base64, and handling local files.
- **Design Choice:** Uses `PIL.ImageOps.exif_transpose` and then `convet("RGB")` for processing the loaded images.

### Notable Dependencies or Requirements
- `requests` (external dependency) for URL requests with timeout
- `PIL` (Python Imaging Library) (required dependency)
- `base64`, `os`, and `BytesIO` (built-in Python modules)

**Method Signatures and Interfaces:**
```python
load_image(str, Optional[float]) -> PIL.Image.Image
```

Note: The RAG system will store this summary for reference.


key: chroma:document
string_value:
CODE:

def load_images(
    images: Union[List, Tuple, str, "PIL.Image.Image"], timeout: Optional[float] = None
) -> Union["PIL.Image.Image", List["PIL.Image.Image"], List[List["PIL.Image.Image"]]]:
    """Loads images, handling different levels of nesting.

    Args:
      images: A single image, a list of images, or a list of lists of images to load.
      timeout: Timeout for loading images.

    Returns:
      A single image, a list of images, a list of lists of images.
    """
    if isinstance(images, (list, tuple)):
        if len(images) and isinstance(images[0], (list, tuple)):
            return [[load_image(image, timeout=timeout) for image in image_group] for image_group in images]
        else:
            return [load_image(image, timeout=timeout) for image in images]
    else:


SUMMARY:
**Function Summary:**

`load_images` loads one or more images from various nested configurations and returns the loaded results.

```
Parameters:
  - `images`: An image (singleton), list of images, or list of lists of images to load.
  - Optional `timeout`

Returns:
  A single image instance, list of images, or list of lists of images
```

**Key Implementation Details:**

* Recursively handles nested configurations using `if isinstance(images, (list, tuple))`
* Uses list comprehension to process different types of input data (`images` and `image_groups`)
* Calls `load_image` recursively to handle each image

**Notable Dependencies/Interfaces:**

* Import from PIL library (`"PIL.Image"` instances)
* Type checking for function arguments and return values
* Optional argument for loading under a timeout


key: chroma:document
string_value:
CODE:

def validate_preprocess_arguments(
    do_rescale: Optional[bool] = None,
    rescale_factor: Optional[float] = None,
    do_normalize: Optional[bool] = None,
    image_mean: Optional[Union[float, List[float]]] = None,
    image_std: Optional[Union[float, List[float]]] = None,
    do_pad: Optional[bool] = None,
    size_divisibility: Optional[int] = None,
    do_center_crop: Optional[bool] = None,
    crop_size: Optional[Dict[str, int]] = None,
    do_resize: Optional[bool] = None,
    size: Optional[Dict[str, int]] = None,
    resample: Optional["PILImageResampling"] = None,
):
    """
    Checks validity of typically used arguments in an `ImageProcessor` `preprocess` method.
    Raises `ValueError` if arguments incompatibility is caught.
    Many incompatibilities are model-specific. `do_pad` sometimes needs `size_divisor`,
    sometimes `size_divisibility`, and sometimes `size`. New models and processors added should follow
    existing arguments when possible.

    """
    if do_rescale and rescale_factor is None:
        raise ValueError("`rescale_factor` must be specified if `do_rescale` is `True`.")

    if do_pad and size_divisibility is None:
        # Here, size_divisor might be passed as the value of size
        raise ValueError(
            "Depending on the model, `size_divisibility`, `size_divisor`, `pad_size` or `size` must be specified if `do_pad` is `True`."
        )

    if do_normalize and (image_mean is None or image_std is None):
        raise ValueError("`image_mean` and `image_std` must both be specified if `do_normalize` is `True`.")

    if do_center_crop and crop_size is None:
        raise ValueError("`crop_size` must be specified if `do_center_crop` is `True`.")

    if do_resize and (size is None or resample is None):


SUMMARY:
**Summary:**

Validate Preprocess Arguments Function:

This function validates the input arguments for an `ImageProcessor` `preprocess` method.

**Purpose:** Validate preprocessing argument incompatibilities, raising `ValueError` if any are found.

**Key Implementation Details:**

* Checks if required parameters are specified when others are set (e.g., `do_rescale` and `/rescale_factor`, `/do_pad` and `/size_divisibility`, etc.).
* Raises `ValueError` for incompatibilities, noting model-specific exceptions.

**Notable Parameters:**

* `/do_rescale`: Rescales input image.
* `/rescale_factor`: Scaling factor for rescaling (optional).
* `/do_normalize`: Normalizes pixel values by adding zero mean and scaling to unit variance.
* `/image_mean` and `/image_std`: Used for normalization if `/do_normalize` is True.
* `/do_pad`: Pads output image with zeros.
* `/size_divisibility`: Size divisibility requirement when pads are specified (optional).
* `/do_center_crop`: Crops center region of input image.
* `/crop_size`: Crop size requirements when center cropping is enabled (optional).
* `/do_resize`: Resizes output image to specified size.
* `/size` and `/resample`: Resize output image if `/do_resize` is True.

**External Dependencies:**

* None explicitly listed, but may depend on PIL library for `PILImageResampling`.


key: chroma:document
string_value:
CODE:

def validate_fast_preprocess_arguments(
    do_rescale: Optional[bool] = None,
    rescale_factor: Optional[float] = None,
    do_normalize: Optional[bool] = None,
    image_mean: Optional[Union[float, List[float]]] = None,
    image_std: Optional[Union[float, List[float]]] = None,
    do_pad: Optional[bool] = None,
    size_divisibility: Optional[int] = None,
    do_center_crop: Optional[bool] = None,
    crop_size: Optional[Dict[str, int]] = None,
    do_resize: Optional[bool] = None,
    size: Optional[Dict[str, int]] = None,
    resample: Optional["PILImageResampling"] = None,
    return_tensors: Optional[Union[str, TensorType]] = None,
    data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
):
    """
    Checks validity of typically used arguments in an `ImageProcessorFast` `preprocess` method.
    Raises `ValueError` if arguments incompatibility is caught.
    """
    validate_preprocess_arguments(
        do_rescale=do_rescale,
        rescale_factor=rescale_factor,
        do_normalize=do_normalize,
        image_mean=image_mean,
        image_std=image_std,
        do_resize=do_resize,
        size=size,
        resample=resample,
    )
    # Extra checks for ImageProcessorFast
    if return_tensors != "pt":
        raise ValueError("Only returning PyTorch tensors is currently supported.")

    if data_format != ChannelDimension.FIRST:


SUMMARY:
**Summary:**

`validate_fast_preprocess_arguments` function validates typical arguments for the `ImageProcessorFast` preprocess method, also known as MobileNet Face Detector. It checks consistency of input parameters and raises `ValueError` if incompatibility is detected.

**Key Implementation Details:**

- Calls `validate_preprocess_statistics` with validation parameters.
- Checks that `return_tensors` equals `"pt"` to support only PyTorch tensors.
- Verifies that `data_format` equals `ChannelDimension.FIRST` for channel-first data format compatibility.

**Notable Dependencies/Requirements:**

* Uses `PILImageResampling`
* Expects `validate_preprocess_arguments`, a separate function not defined in this snippet
* Assumes `TensorType`, `Optional`, and `Union` type hints are correctly implemented.


key: chroma:document
string_value:
CODE:
# In the future we can add a TF implementation here when we have TF models.
class ImageFeatureExtractionMixin:
    """
    Mixin that contain utilities for preparing image features.
    """

    def _ensure_format_supported(self, image):
        if not isinstance(image, (PIL.Image.Image, np.ndarray)) and not is_torch_tensor(image):
            raise ValueError(
                f"Got type {type(image)} which is not supported, only `PIL.Image.Image`, `np.array` and "
                "`torch.Tensor` are."
            )

    def to_pil_image(self, image, rescale=None):
        """
        Converts `image` to a PIL Image. Optionally rescales it and puts the channel dimension back as the last axis if
        needed.

        Args:
            image (`PIL.Image.Image` or `numpy.ndarray` or `torch.Tensor`):
                The image to convert to the PIL Image format.
            rescale (`bool`, *optional*):
                Whether or not to apply the scaling factor (to make pixel values integers between 0 and 255). Will
                default to `True` if the image type is a floating type, `False` otherwise.
        """
        self._ensure_format_supported(image)

        if is_torch_tensor(image):
            image = image.numpy()

        if isinstance(image, np.ndarray):
            if rescale is None:
                # rescale default to the array being of floating type.
                rescale = isinstance(image.flat[0], np.floating)
            # If the channel as been moved to first dim, we put it back at the end.
            if image.ndim == 3 and image.shape[0] in [1, 3]:
                image = image.transpose(1, 2, 0)
            if rescale:
                image = image * 255
            image = image.astype(np.uint8)
            return PIL.Image.fromarray(image)
        return image

    def convert_rgb(self, image):
        """
        Converts `PIL.Image.Image` to RGB format.

        Args:
            image (`PIL.Image.Image`):
                The image to convert.
        """
        self._ensure_format_supported(image)
        if not isinstance(image, PIL.Image.Image):
            return image

        return image.convert("RGB")

    def rescale(self, image: np.ndarray, scale: Union[float, int]) -> np.ndarray:
        """
        Rescale a numpy image by scale amount
        """
        self._ensure_format_supported(image)
        return image * scale

    def to_numpy_array(self, image, rescale=None, channel_first=True):
        """
        Converts `image` to a numpy array. Optionally rescales it and puts the channel dimension as the first
        dimension.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to convert to a NumPy array.
            rescale (`bool`, *optional*):
                Whether or not to apply the scaling factor (to make pixel values floats between 0. and 1.). Will
                default to `True` if the image is a PIL Image or an array/tensor of integers, `False` otherwise.
            channel_first (`bool`, *optional*, defaults to `True`):
                Whether or not to permute the dimensions of the image to put the channel dimension first.
        """
        self._ensure_format_supported(image)

        if isinstance(image, PIL.Image.Image):
            image = np.array(image)

        if is_torch_tensor(image):
            image = image.numpy()

        rescale = isinstance(image.flat[0], np.integer) if rescale is None else rescale

        if rescale:
            image = self.rescale(image.astype(np.float32), 1 / 255.0)

        if channel_first and image.ndim == 3:
            image = image.transpose(2, 0, 1)

        return image

    def expand_dims(self, image):
        """
        Expands 2-dimensional `image` to 3 dimensions.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to expand.
        """
        self._ensure_format_supported(image)

        # Do nothing if PIL image
        if isinstance(image, PIL.Image.Image):
            return image

        if is_torch_tensor(image):
            image = image.unsqueeze(0)
        else:
            image = np.expand_dims(image, axis=0)
        return image

    def normalize(self, image, mean, std, rescale=False):
        """
        Normalizes `image` with `mean` and `std`. Note that this will trigger a conversion of `image` to a NumPy array
        if it's a PIL Image.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to normalize.
            mean (`List[float]` or `np.ndarray` or `torch.Tensor`):
                The mean (per channel) to use for normalization.
            std (`List[float]` or `np.ndarray` or `torch.Tensor`):
                The standard deviation (per channel) to use for normalization.
            rescale (`bool`, *optional*, defaults to `False`):
                Whether or not to rescale the image to be between 0 and 1. If a PIL image is provided, scaling will
                happen automatically.
        """
        self._ensure_format_supported(image)

        if isinstance(image, PIL.Image.Image):
            image = self.to_numpy_array(image, rescale=True)
        # If the input image is a PIL image, it automatically gets rescaled. If it's another
        # type it may need rescaling.
        elif rescale:
            if isinstance(image, np.ndarray):
                image = self.rescale(image.astype(np.float32), 1 / 255.0)
            elif is_torch_tensor(image):
                image = self.rescale(image.float(), 1 / 255.0)

        if isinstance(image, np.ndarray):
            if not isinstance(mean, np.ndarray):
                mean = np.array(mean).astype(image.dtype)
            if not isinstance(std, np.ndarray):
                std = np.array(std).astype(image.dtype)
        elif is_torch_tensor(image):
            import torch

            if not isinstance(mean, torch.Tensor):
                if isinstance(mean, np.ndarray):
                    mean = torch.from_numpy(mean)
                else:
                    mean = torch.tensor(mean)
            if not isinstance(std, torch.Tensor):
                if isinstance(std, np.ndarray):
                    std = torch.from_numpy(std)
                else:
                    std = torch.tensor(std)

        if image.ndim == 3 and image.shape[0] in [1, 3]:
            return (image - mean[:, None, None]) / std[:, None, None]
        else:
            return (image - mean) / std

    def resize(self, image, size, resample=None, default_to_square=True, max_size=None):
        """
        Resizes `image`. Enforces conversion of input to PIL.Image.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to resize.
            size (`int` or `Tuple[int, int]`):
                The size to use for resizing the image. If `size` is a sequence like (h, w), output size will be
                matched to this.

                If `size` is an int and `default_to_square` is `True`, then image will be resized to (size, size). If
                `size` is an int and `default_to_square` is `False`, then smaller edge of the image will be matched to
                this number. i.e, if height > width, then image will be rescaled to (size * height / width, size).
            resample (`int`, *optional*, defaults to `PILImageResampling.BILINEAR`):
                The filter to user for resampling.
            default_to_square (`bool`, *optional*, defaults to `True`):
                How to convert `size` when it is a single int. If set to `True`, the `size` will be converted to a
                square (`size`,`size`). If set to `False`, will replicate
                [`torchvision.transforms.Resize`](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Resize)
                with support for resizing only the smallest edge and providing an optional `max_size`.
            max_size (`int`, *optional*, defaults to `None`):
                The maximum allowed for the longer edge of the resized image: if the longer edge of the image is
                greater than `max_size` after being resized according to `size`, then the image is resized again so
                that the longer edge is equal to `max_size`. As a result, `size` might be overruled, i.e the smaller
                edge may be shorter than `size`. Only used if `default_to_square` is `False`.

        Returns:
            image: A resized `PIL.Image.Image`.
        """
        resample = resample if resample is not None else PILImageResampling.BILINEAR

        self._ensure_format_supported(image)

        if not isinstance(image, PIL.Image.Image):
            image = self.to_pil_image(image)

        if isinstance(size, list):
            size = tuple(size)

        if isinstance(size, int) or len(size) == 1:
            if default_to_square:
                size = (size, size) if isinstance(size, int) else (size[0], size[0])
            else:
                width, height = image.size
                # specified size only for the smallest edge
                short, long = (width, height) if width <= height else (height, width)
                requested_new_short = size if isinstance(size, int) else size[0]

                if short == requested_new_short:
                    return image

                new_short, new_long = requested_new_short, int(requested_new_short * long / short)

                if max_size is not None:
                    if max_size <= requested_new_short:
                        raise ValueError(
                            f"max_size = {max_size} must be strictly greater than the requested "
                            f"size for the smaller edge size = {size}"
                        )
                    if new_long > max_size:
                        new_short, new_long = int(max_size * new_short / new_long), max_size

                size = (new_short, new_long) if width <= height else (new_long, new_short)

        return image.resize(size, resample=resample)

    def center_crop(self, image, size):
        """
        Crops `image` to the given size using a center crop. Note that if the image is too small to be cropped to the
        size given, it will be padded (so the returned result has the size asked).

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor` of shape (n_channels, height, width) or (height, width, n_channels)):
                The image to resize.
            size (`int` or `Tuple[int, int]`):
                The size to which crop the image.

        Returns:
            new_image: A center cropped `PIL.Image.Image` or `np.ndarray` or `torch.Tensor` of shape: (n_channels,
            height, width).
        """
        self._ensure_format_supported(image)

        if not isinstance(size, tuple):
            size = (size, size)

        # PIL Image.size is (width, height) but NumPy array and torch Tensors have (height, width)
        if is_torch_tensor(image) or isinstance(image, np.ndarray):
            if image.ndim == 2:
                image = self.expand_dims(image)
            image_shape = image.shape[1:] if image.shape[0] in [1, 3] else image.shape[:2]
        else:
            image_shape = (image.size[1], image.size[0])

        top = (image_shape[0] - size[0]) // 2
        bottom = top + size[0]  # In case size is odd, (image_shape[0] + size[0]) // 2 won't give the proper result.
        left = (image_shape[1] - size[1]) // 2
        right = left + size[1]  # In case size is odd, (image_shape[1] + size[1]) // 2 won't give the proper result.

        # For PIL Images we have a method to crop directly.
        if isinstance(image, PIL.Image.Image):
            return image.crop((left, top, right, bottom))

        # Check if image is in (n_channels, height, width) or (height, width, n_channels) format
        channel_first = True if image.shape[0] in [1, 3] else False

        # Transpose (height, width, n_channels) format images
        if not channel_first:
            if isinstance(image, np.ndarray):
                image = image.transpose(2, 0, 1)
            if is_torch_tensor(image):
                image = image.permute(2, 0, 1)

        # Check if cropped area is within image boundaries
        if top >= 0 and bottom <= image_shape[0] and left >= 0 and right <= image_shape[1]:
            return image[..., top:bottom, left:right]

        # Otherwise, we may need to pad if the image is too small. Oh joy...
        new_shape = image.shape[:-2] + (max(size[0], image_shape[0]), max(size[1], image_shape[1]))
        if isinstance(image, np.ndarray):
            new_image = np.zeros_like(image, shape=new_shape)
        elif is_torch_tensor(image):
            new_image = image.new_zeros(new_shape)

        top_pad = (new_shape[-2] - image_shape[0]) // 2
        bottom_pad = top_pad + image_shape[0]
        left_pad = (new_shape[-1] - image_shape[1]) // 2
        right_pad = left_pad + image_shape[1]
        new_image[..., top_pad:bottom_pad, left_pad:right_pad] = image

        top += top_pad
        bottom += top_pad
        left += left_pad
        right += left_pad

        new_image = new_image[
            ..., max(0, top) : min(new_image.shape[-2], bottom), max(0, left) : min(new_image.shape[-1], right)
        ]

        return new_image

    def flip_channel_order(self, image):
        """
        Flips the channel order of `image` from RGB to BGR, or vice versa. Note that this will trigger a conversion of
        `image` to a NumPy array if it's a PIL Image.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image whose color channels to flip. If `np.ndarray` or `torch.Tensor`, the channel dimension should
                be first.
        """
        self._ensure_format_supported(image)

        if isinstance(image, PIL.Image.Image):
            image = self.to_numpy_array(image)

        return image[::-1, :, :]

    def rotate(self, image, angle, resample=None, expand=0, center=None, translate=None, fillcolor=None):
        """
        Returns a rotated copy of `image`. This method returns a copy of `image`, rotated the given number of degrees
        counter clockwise around its centre.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to rotate. If `np.ndarray` or `torch.Tensor`, will be converted to `PIL.Image.Image` before
                rotating.

        Returns:
            image: A rotated `PIL.Image.Image`.
        """
        resample = resample if resample is not None else PIL.Image.NEAREST

        self._ensure_format_supported(image)

        if not isinstance(image, PIL.Image.Image):
            image = self.to_pil_image(image)

        return image.rotate(
            angle, resample=resample, expand=expand, center=center, translate=translate, fillcolor=fillcolor


SUMMARY:
It is an object-oriented class in Python for image processing. The main functionalities include resizing, flipping channel order, and rotating images.

Here are some example use cases:

**Resize**
```python
from PIL import Image

img = Image.open('image.jpg')
resized_img = obj.resize((800, 600))
```

**Flip Channel Order**
```python
from PIL import Image

img = Image.open('image.jpg')
flipped_img = obj.flip_channel_order(img)
```

**Rotate**
```python
from PIL import Image

img = Image.open('image.jpg')
rotated_img = obj.rotate(90, resample=PIL.Image.NEAREST)
```

The class appears to be well-structured and readable, with clear docstrings for each method. It also uses the Pillow library, which is a popular choice for image processing in Python.

However, there are a few suggestions that can improve the code:

* Some of the variable names could be more descriptive. For example, `top`, `bottom`, `left`, and `right` in the `flip_channel_order` method could be renamed to something like `height_offset` and `width_offset`.
* The class does not seem to have any methods for applying transformations to PIL Images. This could be added as a separate class with its own docstrings.
* Some of the constants, such as `PIL.Image.NEAREST`, could be moved outside the class or made into properties if they are used in multiple places.

Overall, the code is well-written and easy to understand. With some minor improvements, it could be even more readable and maintainable.


key: chroma:document
string_value:
CODE:

def validate_annotations(
    annotation_format: AnnotationFormat,
    supported_annotation_formats: Tuple[AnnotationFormat, ...],
    annotations: List[Dict],
) -> None:
    if annotation_format not in supported_annotation_formats:
        raise ValueError(f"Unsupported annotation format: {format} must be one of {supported_annotation_formats}")

    if annotation_format is AnnotationFormat.COCO_DETECTION:
        if not valid_coco_detection_annotations(annotations):
            raise ValueError(
                "Invalid COCO detection annotations. Annotations must a dict (single image) or list of dicts "
                "(batch of images) with the following keys: `image_id` and `annotations`, with the latter "
                "being a list of annotations in the COCO format."
            )

    if annotation_format is AnnotationFormat.COCO_PANOPTIC:
        if not valid_coco_panoptic_annotations(annotations):
            raise ValueError(
                "Invalid COCO panoptic annotations. Annotations must a dict (single image) or list of dicts "
                "(batch of images) with the following keys: `image_id`, `file_name` and `segments_info`, with "
                "the latter being a list of annotations in the COCO format."


SUMMARY:
**Function Summary**

`validate_annotations`: Validates input annotations against supported formats, raising errors for incompatible or malformed data.

```markdown
### Purpose and Functionality

* Validates `annotation_format` against `supported_annotation_formats`.
	+ Checks for supported format: `AnnotationFormat.annotation_format in supported_annotation_formats`
* Performs format-specific validation:
	+ COCO Detection: Validates `annotations` as either a dictionary (single image) or list of dictionaries (batch of images) with required keys.
	+ COCO Panoptic: Validates `annotations` as a dictionary (single image) or list of dictionaries with additional panoptic annotations.

### Implementation Details

* Uses functions:
	+ `valid_coco_detection_annotations`
	+ `valid_coco_panoptic_annotations`
* Raises errors for invalid formats, missing required keys, or incorrect annotation structure.
```

**Notable Dependencies**

* External dependencies: None
* Inherited interfaces:
	+ `AnnotationFormat`: supported in `supported_annotation_formats` tuple
	+ `Dict`, `List`, `Tuple`: used as return types and parameter types

**Key Method Signatures and Interfaces**

* `validate_annotations(annotation_format, supported_annotation_formats, annotations) -> None`
	+ Validates input annotations against supported formats.
* `valid_coco_detection_annotations(annotations) -> bool`
	+ Validates COCO detection annotations.
* `valid_coco_panoptic_annotations(annotations) -> bool`
	+ Validates COCO panoptic annotations.


key: chroma:document
string_value:
CODE:

def validate_kwargs(valid_processor_keys: List[str], captured_kwargs: List[str]):
    unused_keys = set(captured_kwargs).difference(set(valid_processor_keys))
    if unused_keys:
        unused_key_str = ", ".join(unused_keys)
        # TODO raise a warning here instead of simply logging?


SUMMARY:
**Function Summary**

`validate_kwargs` validates argument keywords against a predefined set of valid processor keys. It checks for unused or unrecognized keyword arguments and logs warnings.

**Key Implementation Details**

* Checks for and removes unused `captured_kwargs` not present in `valid_processor_keys`.
* Logs a warning when an unused key is found, using the joined string of all unused keys.
* Current implementation does not raise an exception; instead, it logs a warning.

**Notable Dependencies/Requirements**

* `List[str]`: The typesafe type hint for the input parameters.
* `logger` (assuming a logging module, e.g., Python's built-in `logging`): Used to log warnings about unused or unrecognized keywords.


key: chroma:document
string_value:
CODE:

class BaseImageProcessorFast(BaseImageProcessor):
    _transform_params = None

    def _build_transforms(self, **kwargs) -> "Compose":
        """
        Given the input settings e.g. do_resize, build the image transforms.
        """
        raise NotImplementedError

    def _validate_params(self, **kwargs) -> None:
        for k, v in kwargs.items():
            if k not in self._transform_params:
                raise ValueError(f"Invalid transform parameter {k}={v}.")

    @functools.lru_cache(maxsize=1)
    def get_transforms(self, **kwargs) -> "Compose":
        self._validate_params(**kwargs)
        return self._build_transforms(**kwargs)

    def to_dict(self):
        encoder_dict = super().to_dict()
        encoder_dict.pop("_transform_params", None)


SUMMARY:
**Summary**

The `BaseImageProcessorFast` class extends `BaseImageProcessor`. It serves as a base class for fast image processors, implementing the following key functionalities:

*   Validates input parameters against predefined transforms (_validate_params)
*   Builds image transforms based on input settings (_build_transforms)
*   Caches transform builds using memoization (get_transforms)
*   Converts an object to a dictionary (to_dict)

**Notable Details**

*   The `BaseImageProcessorFast` class reuses `_transform_params`, which is initialized as empty.
*   It inherits from `_build_transforms` method, leaving it unimplemented, indicating that it should be overridden by subclasses for specific transforms.
*   The `get_transforms` method relies on memoization to cache the results of `_build_transforms`.
*   The `to_dict` method removes an unsupported field (`_transform_params`) from the object's dictionary representation.

**Dependencies and Interfaces**

*   Extends `BaseImageProcessor`
*   Relies on private methods in `BaseImageProcessor`
*   None


key: chroma:document
string_value:
CODE:

def get_image_size_for_max_height_width(
    image_size: Tuple[int, int],
    max_height: int,
    max_width: int,
) -> Tuple[int, int]:
    """
    Computes the output image size given the input image and the maximum allowed height and width. Keep aspect ratio.
    Important, even if image_height < max_height and image_width < max_width, the image will be resized
    to at least one of the edges be equal to max_height or max_width.

    For example:
        - input_size: (100, 200), max_height: 50, max_width: 50 -> output_size: (25, 50)
        - input_size: (100, 200), max_height: 200, max_width: 500 -> output_size: (200, 400)

    Args:
        image_size (`Tuple[int, int]`):
            The image to resize.
        max_height (`int`):
            The maximum allowed height.
        max_width (`int`):
            The maximum allowed width.
    """
    height, width = image_size
    height_scale = max_height / height
    width_scale = max_width / width
    min_scale = min(height_scale, width_scale)
    new_height = int(height * min_scale)
    new_width = int(width * min_scale)


SUMMARY:
**Summary:**

`get_image_size_for_max_height_width` function calculates the output image size while maintaining the aspect ratio. Given input `image_size`, `max_height`, and `max_width`, it returns the smallest scaled dimensions to fit within the specified limits.

**Key Implementation Details:**

* Calculates scaling factors for each dimension (`height_scale`, `width_scale`)
* Computes the minimum scale factor (`min_scale`) that satisfies both constraints
* Applies this scale factor to calculate new dimensions (`new_height`, `new_width`)

**Notable Dependencies or Requirements:**

* Expects input as a tuple of two integers (image size)
* Inherently scales image based on aspect ratio preservation.


key: chroma:document
string_value:
CODE:

def safe_squeeze(tensor: "torch.Tensor", axis: Optional[int] = None) -> "torch.Tensor":
    """
    Squeezes a tensor, but only if the axis specified has dim 1.
    """
    if axis is None:
        return tensor.squeeze()

    try:
        return tensor.squeeze(axis=axis)
    except ValueError:


SUMMARY:
**Summary:**

`safe_squeeze` Function:

* Squeezes a PyTorch tensor along the specified axis (defaulting to all axes except 0 if no axis is provided).
* Exceptions are caught and swallowed by returning the original tensor, avoiding `ValueError`.

**Key Details:**

* `tensor.squeeze(axis=axis)` with defensive error handling.
* Axes 1-9 can be squeezed; axes 0 and other special axes not supported.

**Notable Dependencies:**

* PyTorch (`torch.Tensor`)
* Python's built-in `Optional` type for type hinting.


key: chroma:document
string_value:
CODE:

def max_across_indices(values: Iterable[Any]) -> List[Any]:
    """
    Return the maximum value across all indices of an iterable of values.
    """


SUMMARY:
**Summary**

```
**Functionality**: Returns the maximum value across all indices of an iterable of values.

**Implementation Details**:

* Uses `zip(*values)` to transpose and unpack the input iterable into separate lists for each index.
* Applies the `max` function on each of these lists using a list comprehension.

**Dependencies/Requirements**:
 None.
 
 Note: This summary only includes essential information about the code's functionality, implementation details, and dependencies. It adheres to RAG system guidelines by preserving exact class/function names, highlighting key algorithms (zip/unpacking), and omitting unnecessary details.


key: chroma:document
string_value:
CODE:

def get_max_height_width(images: List["torch.Tensor"]) -> Tuple[int]:
    """
    Get the maximum height and width across all images in a batch.
    """

    _, max_height, max_width = max_across_indices([img.shape for img in images])



SUMMARY:
**Function Summary: get_max_height_width**

Retrieves the maximum height and width from a batch of images using `torch.Tensor`.

### Implementation Details:

* Calculates maximum dimensions across image batches using `max_across_indices`.
* Returns a tuple containing the maximum height and width.

### Dependencies/Requirements:

* `torch`
* `List` and `Tuple` from Python's types module (`typing`)

### Context:

This function appears to be part of an image processing pipeline, suggesting it may be used in computer vision or machine learning applications.


key: chroma:document
string_value:
CODE:
# From https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse
def string_to_bool(v):
    if isinstance(v, bool):
        return v
    if v.lower() in ("yes", "true", "t", "y", "1"):
        return True
    elif v.lower() in ("no", "false", "f", "n", "0"):
        return False
    else:
        raise ArgumentTypeError(
            f"Truthy value expected: got {v} but expected one of yes/no, true/false, t/f, y/n, 1/0 (case insensitive)."


SUMMARY:
**Function Summary**

`string_to_bool`
================

Converts a string to its equivalent boolean value.

### Purpose
Parses a string into a boolean value, with optional case-insensitive conversion.

### Implementation Details

* Checks for explicit `bool` value
* Converts input strings to lowercase and checks for specified truthy values (yes/true/t/y/1) or falsey values (no/false/f/n/0)
* Raises `ArgumentTypeError` if invalid value is provided

### Notable Dependencies

* `isinstance`
* `str.lower`

### Method Signature
```
def string_to_bool(v: object) -> bool:
```


key: chroma:document
string_value:
CODE:

def make_choice_type_function(choices: list) -> Callable[[str], Any]:
    """
    Creates a mapping function from each choices string representation to the actual value. Used to support multiple
    value types for a single argument.

    Args:
        choices (list): List of choices.

    Returns:
        Callable[[str], Any]: Mapping function from string representation to actual value for each choice.
    """
    str_to_choice = {str(choice): choice for choice in choices}


SUMMARY:
**Summary:**

Creates a mapping function `make_choice_type_function` that maps string representations of choices to their actual values. This function uses a dictionary to store the mappings and returns a lambda function that looks up the choice value based on the input string representation.

**Key Implementation Details:**

* Maps each choice string to its corresponding value using a dictionary comprehension.
* Returns a lambda function with the signature `lambda arg: str_to_choice.get(arg, arg)`.

**Notable Dependencies or Requirements:**

* Uses the `Callable` type hint and the `Any` return type hint from the Python typing library.
* Inherit/Implements: No external interfaces are implemented. However, it uses the `list` type for the `choices` argument and returns a function with the `Callable[[str], Any]` signature.

**Context:** This code is likely part of a larger testing or validation framework to support multiple value types for a single argument.


key: chroma:document
string_value:
CODE:

def HfArg(
    *,
    aliases: Union[str, List[str]] = None,
    help: str = None,
    default: Any = dataclasses.MISSING,
    default_factory: Callable[[], Any] = dataclasses.MISSING,
    metadata: dict = None,
    **kwargs,
) -> dataclasses.Field:
    """Argument helper enabling a concise syntax to create dataclass fields for parsing with `HfArgumentParser`.

    Example comparing the use of `HfArg` and `dataclasses.field`:
    ```
    @dataclass
    class Args:
        regular_arg: str = dataclasses.field(default="Huggingface", metadata={"aliases": ["--example", "-e"], "help": "This syntax could be better!"})
        hf_arg: str = HfArg(default="Huggingface", aliases=["--example", "-e"], help="What a nice syntax!")
    ```

    Args:
        aliases (Union[str, List[str]], optional):
            Single string or list of strings of aliases to pass on to argparse, e.g. `aliases=["--example", "-e"]`.
            Defaults to None.
        help (str, optional): Help string to pass on to argparse that can be displayed with --help. Defaults to None.
        default (Any, optional):
            Default value for the argument. If not default or default_factory is specified, the argument is required.
            Defaults to dataclasses.MISSING.
        default_factory (Callable[[], Any], optional):
            The default_factory is a 0-argument function called to initialize a field's value. It is useful to provide
            default values for mutable types, e.g. lists: `default_factory=list`. Mutually exclusive with `default=`.
            Defaults to dataclasses.MISSING.
        metadata (dict, optional): Further metadata to pass on to `dataclasses.field`. Defaults to None.

    Returns:
        Field: A `dataclasses.Field` with the desired properties.
    """
    if metadata is None:
        # Important, don't use as default param in function signature because dict is mutable and shared across function calls
        metadata = {}
    if aliases is not None:
        metadata["aliases"] = aliases
    if help is not None:
        metadata["help"] = help



SUMMARY:
**Summary: `HfArg` Function**

The `HfArg` function provides a concise syntax to create dataclass fields for parsing with `HfArgumentParser`. It enables creating required or optional arguments with customizable aliases, help messages, and default values.

**Key Features:**

*   Creates a `dataclasses.Field` object with customizable properties:
    *   `aliases`: A list of string aliases to pass on to argparse.
    *   `help`: The help message to be displayed with `--help`.
    *   `default`: The default value for the argument.
    *   `default_factory`: A function called to initialize a field's value (mutually exclusive with `default`).
    *   Additional metadata (`**kwargs`).

**Method Signature:**
```python
def HfArg(
    *,
    aliases: Union[str, List[str]] = None,
    help: str = None,
    default: Any = dataclasses.MISSING,
    default_factory: Callable[[], Any] = dataclasses.Missing,
    metadata: dict = None,
    **kwargs,
) -> dataclasses.Field:
```

**Key Data Types and Functions Used:**

*   `Union[str, List[str]]`: Union type for aliases.
*   `Callable[[], Any]`: Function signature for `default_factory`.
*   `dataclasses.Field`: The class returned by `HfArg`.

This function provides a convenient way to create dataclass fields with customized properties, making it easier to parse arguments with `HfArgumentParser`.


key: chroma:document
string_value:
CODE:

class HfArgumentParser(ArgumentParser):
    """
    This subclass of `argparse.ArgumentParser` uses type hints on dataclasses to generate arguments.

    The class is designed to play well with the native argparse. In particular, you can add more (non-dataclass backed)
    arguments to the parser after initialization and you'll get the output back after parsing as an additional
    namespace. Optional: To create sub argument groups use the `_argument_group_name` attribute in the dataclass.
    """

    dataclass_types: Iterable[DataClassType]

    def __init__(self, dataclass_types: Union[DataClassType, Iterable[DataClassType]], **kwargs):
        """
        Args:
            dataclass_types:
                Dataclass type, or list of dataclass types for which we will "fill" instances with the parsed args.
            kwargs (`Dict[str, Any]`, *optional*):
                Passed to `argparse.ArgumentParser()` in the regular way.
        """
        # To make the default appear when using --help
        if "formatter_class" not in kwargs:
            kwargs["formatter_class"] = ArgumentDefaultsHelpFormatter
        super().__init__(**kwargs)
        if dataclasses.is_dataclass(dataclass_types):
            dataclass_types = [dataclass_types]
        self.dataclass_types = list(dataclass_types)
        for dtype in self.dataclass_types:
            self._add_dataclass_arguments(dtype)

    @staticmethod
    def _parse_dataclass_field(parser: ArgumentParser, field: dataclasses.Field):
        # Long-option strings are conventionlly separated by hyphens rather
        # than underscores, e.g., "--long-format" rather than "--long_format".
        # Argparse converts hyphens to underscores so that the destination
        # string is a valid attribute name. Hf_argparser should do the same.
        long_options = [f"--{field.name}"]
        if "_" in field.name:
            long_options.append(f"--{field.name.replace('_', '-')}")

        kwargs = field.metadata.copy()
        # field.metadata is not used at all by Data Classes,
        # it is provided as a third-party extension mechanism.
        if isinstance(field.type, str):
            raise RuntimeError(
                "Unresolved type detected, which should have been done with the help of "
                "`typing.get_type_hints` method by default"
            )

        aliases = kwargs.pop("aliases", [])
        if isinstance(aliases, str):
            aliases = [aliases]

        origin_type = getattr(field.type, "__origin__", field.type)
        if origin_type is Union or (hasattr(types, "UnionType") and isinstance(origin_type, types.UnionType)):
            if str not in field.type.__args__ and (
                len(field.type.__args__) != 2 or type(None) not in field.type.__args__
            ):
                raise ValueError(
                    "Only `Union[X, NoneType]` (i.e., `Optional[X]`) is allowed for `Union` because"
                    " the argument parser only supports one type per argument."
                    f" Problem encountered in field '{field.name}'."
                )
            if type(None) not in field.type.__args__:
                # filter `str` in Union
                field.type = field.type.__args__[0] if field.type.__args__[1] is str else field.type.__args__[1]
                origin_type = getattr(field.type, "__origin__", field.type)
            elif bool not in field.type.__args__:
                # filter `NoneType` in Union (except for `Union[bool, NoneType]`)
                field.type = (
                    field.type.__args__[0] if isinstance(None, field.type.__args__[1]) else field.type.__args__[1]
                )
                origin_type = getattr(field.type, "__origin__", field.type)

        # A variable to store kwargs for a boolean field, if needed
        # so that we can init a `no_*` complement argument (see below)
        bool_kwargs = {}
        if origin_type is Literal or (isinstance(field.type, type) and issubclass(field.type, Enum)):
            if origin_type is Literal:
                kwargs["choices"] = field.type.__args__
            else:
                kwargs["choices"] = [x.value for x in field.type]

            kwargs["type"] = make_choice_type_function(kwargs["choices"])

            if field.default is not dataclasses.MISSING:
                kwargs["default"] = field.default
            else:
                kwargs["required"] = True
        elif field.type is bool or field.type == Optional[bool]:
            # Copy the currect kwargs to use to instantiate a `no_*` complement argument below.
            # We do not initialize it here because the `no_*` alternative must be instantiated after the real argument
            bool_kwargs = copy(kwargs)

            # Hack because type=bool in argparse does not behave as we want.
            kwargs["type"] = string_to_bool
            if field.type is bool or (field.default is not None and field.default is not dataclasses.MISSING):
                # Default value is False if we have no default when of type bool.
                default = False if field.default is dataclasses.MISSING else field.default
                # This is the value that will get picked if we don't include --{field.name} in any way
                kwargs["default"] = default
                # This tells argparse we accept 0 or 1 value after --{field.name}
                kwargs["nargs"] = "?"
                # This is the value that will get picked if we do --{field.name} (without value)
                kwargs["const"] = True
        elif isclass(origin_type) and issubclass(origin_type, list):
            kwargs["type"] = field.type.__args__[0]
            kwargs["nargs"] = "+"
            if field.default_factory is not dataclasses.MISSING:
                kwargs["default"] = field.default_factory()
            elif field.default is dataclasses.MISSING:
                kwargs["required"] = True
        else:
            kwargs["type"] = field.type
            if field.default is not dataclasses.MISSING:
                kwargs["default"] = field.default
            elif field.default_factory is not dataclasses.MISSING:
                kwargs["default"] = field.default_factory()
            else:
                kwargs["required"] = True
        parser.add_argument(*long_options, *aliases, **kwargs)

        # Add a complement `no_*` argument for a boolean field AFTER the initial field has already been added.
        # Order is important for arguments with the same destination!
        # We use a copy of earlier kwargs because the original kwargs have changed a lot before reaching down
        # here and we do not need those changes/additional keys.
        if field.default is True and (field.type is bool or field.type == Optional[bool]):
            bool_kwargs["default"] = False
            parser.add_argument(
                f"--no_{field.name}",
                f"--no-{field.name.replace('_', '-')}",
                action="store_false",
                dest=field.name,
                **bool_kwargs,
            )

    def _add_dataclass_arguments(self, dtype: DataClassType):
        if hasattr(dtype, "_argument_group_name"):
            parser = self.add_argument_group(dtype._argument_group_name)
        else:
            parser = self

        try:
            type_hints: Dict[str, type] = get_type_hints(dtype)
        except NameError:
            raise RuntimeError(
                f"Type resolution failed for {dtype}. Try declaring the class in global scope or "
                "removing line of `from __future__ import annotations` which opts in Postponed "
                "Evaluation of Annotations (PEP 563)"
            )
        except TypeError as ex:
            # Remove this block when we drop Python 3.9 support
            if sys.version_info[:2] < (3, 10) and "unsupported operand type(s) for |" in str(ex):
                python_version = ".".join(map(str, sys.version_info[:3]))
                raise RuntimeError(
                    f"Type resolution failed for {dtype} on Python {python_version}. Try removing "
                    "line of `from __future__ import annotations` which opts in union types as "
                    "`X | Y` (PEP 604) via Postponed Evaluation of Annotations (PEP 563). To "
                    "support Python versions that lower than 3.10, you need to use "
                    "`typing.Union[X, Y]` instead of `X | Y` and `typing.Optional[X]` instead of "
                    "`X | None`."
                ) from ex
            raise

        for field in dataclasses.fields(dtype):
            if not field.init:
                continue
            field.type = type_hints[field.name]
            self._parse_dataclass_field(parser, field)

    def parse_args_into_dataclasses(
        self,
        args=None,
        return_remaining_strings=False,
        look_for_args_file=True,
        args_filename=None,
        args_file_flag=None,
    ) -> Tuple[DataClass, ...]:
        """
        Parse command-line args into instances of the specified dataclass types.

        This relies on argparse's `ArgumentParser.parse_known_args`. See the doc at:
        docs.python.org/3.7/library/argparse.html#argparse.ArgumentParser.parse_args

        Args:
            args:
                List of strings to parse. The default is taken from sys.argv. (same as argparse.ArgumentParser)
            return_remaining_strings:
                If true, also return a list of remaining argument strings.
            look_for_args_file:
                If true, will look for a ".args" file with the same base name as the entry point script for this
                process, and will append its potential content to the command line args.
            args_filename:
                If not None, will uses this file instead of the ".args" file specified in the previous argument.
            args_file_flag:
                If not None, will look for a file in the command-line args specified with this flag. The flag can be
                specified multiple times and precedence is determined by the order (last one wins).

        Returns:
            Tuple consisting of:

                - the dataclass instances in the same order as they were passed to the initializer.abspath
                - if applicable, an additional namespace for more (non-dataclass backed) arguments added to the parser
                  after initialization.
                - The potential list of remaining argument strings. (same as argparse.ArgumentParser.parse_known_args)
        """

        if args_file_flag or args_filename or (look_for_args_file and len(sys.argv)):
            args_files = []

            if args_filename:
                args_files.append(Path(args_filename))
            elif look_for_args_file and len(sys.argv):
                args_files.append(Path(sys.argv[0]).with_suffix(".args"))

            # args files specified via command line flag should overwrite default args files so we add them last
            if args_file_flag:
                # Create special parser just to extract the args_file_flag values
                args_file_parser = ArgumentParser()
                args_file_parser.add_argument(args_file_flag, type=str, action="append")

                # Use only remaining args for further parsing (remove the args_file_flag)
                cfg, args = args_file_parser.parse_known_args(args=args)
                cmd_args_file_paths = vars(cfg).get(args_file_flag.lstrip("-"), None)

                if cmd_args_file_paths:
                    args_files.extend([Path(p) for p in cmd_args_file_paths])

            file_args = []
            for args_file in args_files:
                if args_file.exists():
                    file_args += args_file.read_text().split()

            # in case of duplicate arguments the last one has precedence
            # args specified via the command line should overwrite args from files, so we add them last
            args = file_args + args if args is not None else file_args + sys.argv[1:]
        namespace, remaining_args = self.parse_known_args(args=args)
        outputs = []
        for dtype in self.dataclass_types:
            keys = {f.name for f in dataclasses.fields(dtype) if f.init}
            inputs = {k: v for k, v in vars(namespace).items() if k in keys}
            for k in keys:
                delattr(namespace, k)
            obj = dtype(**inputs)
            outputs.append(obj)
        if len(namespace.__dict__) > 0:
            # additional namespace.
            outputs.append(namespace)
        if return_remaining_strings:
            return (*outputs, remaining_args)
        else:
            if remaining_args:
                raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")

            return (*outputs,)

    def parse_dict(self, args: Dict[str, Any], allow_extra_keys: bool = False) -> Tuple[DataClass, ...]:
        """
        Alternative helper method that does not use `argparse` at all, instead uses a dict and populating the dataclass
        types.

        Args:
            args (`dict`):
                dict containing config values
            allow_extra_keys (`bool`, *optional*, defaults to `False`):
                Defaults to False. If False, will raise an exception if the dict contains keys that are not parsed.

        Returns:
            Tuple consisting of:

                - the dataclass instances in the same order as they were passed to the initializer.
        """
        unused_keys = set(args.keys())
        outputs = []
        for dtype in self.dataclass_types:
            keys = {f.name for f in dataclasses.fields(dtype) if f.init}
            inputs = {k: v for k, v in args.items() if k in keys}
            unused_keys.difference_update(inputs.keys())
            obj = dtype(**inputs)
            outputs.append(obj)
        if not allow_extra_keys and unused_keys:
            raise ValueError(f"Some keys are not used by the HfArgumentParser: {sorted(unused_keys)}")
        return tuple(outputs)

    def parse_json_file(
        self, json_file: Union[str, os.PathLike], allow_extra_keys: bool = False
    ) -> Tuple[DataClass, ...]:
        """
        Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the
        dataclass types.

        Args:
            json_file (`str` or `os.PathLike`):
                File name of the json file to parse
            allow_extra_keys (`bool`, *optional*, defaults to `False`):
                Defaults to False. If False, will raise an exception if the json file contains keys that are not
                parsed.

        Returns:
            Tuple consisting of:

                - the dataclass instances in the same order as they were passed to the initializer.
        """
        with open(Path(json_file), encoding="utf-8") as open_json_file:
            data = json.loads(open_json_file.read())
        outputs = self.parse_dict(data, allow_extra_keys=allow_extra_keys)
        return tuple(outputs)

    def parse_yaml_file(
        self, yaml_file: Union[str, os.PathLike], allow_extra_keys: bool = False
    ) -> Tuple[DataClass, ...]:
        """
        Alternative helper method that does not use `argparse` at all, instead loading a yaml file and populating the
        dataclass types.

        Args:
            yaml_file (`str` or `os.PathLike`):
                File name of the yaml file to parse
            allow_extra_keys (`bool`, *optional*, defaults to `False`):
                Defaults to False. If False, will raise an exception if the json file contains keys that are not
                parsed.

        Returns:
            Tuple consisting of:

                - the dataclass instances in the same order as they were passed to the initializer.
        """
        outputs = self.parse_dict(yaml.safe_load(Path(yaml_file).read_text()), allow_extra_keys=allow_extra_keys)


SUMMARY:
This is an implementation of a Python class called `HfArgumentParser` that serves as a parser for command-line arguments and configuration files in the context of Hugging Face's Hub. The class provides several methods for parsing different formats, including `argparse`, JSON, YAML, and a dictionary.

Here are some key features:

*   **Supports multiple input formats**: The `HfArgumentParser` supports parsing from `argparse` objects, as well as CSV and dictionaries.
*   **Configuration file support**: It can parse configuration files in JSON and YAML format.
*   **Dataclass support**: It provides a method to create instances of dataclasses using the parsed values.

Some potential modifications or improvements could be:

*   **Error handling**: Adding more specific error messages for different types of input errors would improve user experience.
*   **Config file validation**: While JSON and YAML files can contain extra keys, the parser currently allows them. Introducing optional schema validation to enforce strict adherence to expected dataclass attributes might enhance robustness.
*   **Type hints for method parameters**: Adding type hints could improve code readability by specifying the types of all input parameters.

This class appears to be designed as a utility within an NLP or machine learning workflow, possibly with tools for working on specific tasks like model training or inference.


key: chroma:document
string_value:
CODE:

class PytorchGELUTanh(nn.Module):
    """
    A fast C implementation of the tanh approximation of the GeLU activation function. See
    https://arxiv.org/abs/1606.08415.

    This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical
    match due to rounding errors.
    """

    def __init__(self):
        super().__init__()
        if version.parse(torch.__version__) < version.parse("1.12.0"):
            raise ImportError(
                f"You are using torch=={torch.__version__}, but torch>=1.12.0 is required to use "
                "PytorchGELUTanh. Please upgrade torch."
            )

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**PytorchGELUTanh Class Summary**

**Main Purpose and Functionality**
The PytorchGELUTanh class implements the tanh approximation of the GeLU activation function in PyTorch, providing a fast and optimized alternative to NewGELU and FastGELU.

**Key Implementation Details**

* Uses `nn.functional.gelu` with `approximate="tanh"` to simulate GeLU behavior similar to NewGELU and FastGELU.
* Rounding errors introduce non-exact numerical matching.

**Notable Dependencies and Requirements**

* Requires `torch>=1.12.0` for functionality; version less than 1.12.0 will raise an error.
* Uses `nn.functional.gelu` from PyTorch functional module.


key: chroma:document
string_value:
CODE:

class NewGELUActivation(nn.Module):
    """
    Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see
    the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415
    """

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**Summary:**

The `NewGELUActivation` class implements the GELU (Gaussian Error Linear Unit) activation function from the paper "Gaussian Error Linear Units" by Hendrycks and Gimpel (2016). The forward method takes an input tensor, applies element-wise operations to compute a scaled and shifted version of the input, which is then passed through a sigmoid function.

**Key Implementation Details:**

* Applies scaling factor `sqrt(2.0 / pi)` to input to stabilize gradients
* Employs cubing (`torch.pow(input, 3.0)`) for robustness improvement
* Incorporates sigmoid (`torch.tanh` with inverse hyperbolic sine calculation) to transition from linear and ReLU-like behavior

**Notable Dependencies:**

* `torch`: PyTorch library for tensor computations and activation functions
* `math`: Python standard library for mathematical operations (sqrt, pi)


key: chroma:document
string_value:
CODE:

class GELUActivation(nn.Module):
    """
    Original Implementation of the GELU activation function in Google BERT repo when initially created. For
    information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 +
    torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional
    Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415
    """

    def __init__(self, use_gelu_python: bool = False):
        super().__init__()
        if use_gelu_python:
            self.act = self._gelu_python
        else:
            self.act = nn.functional.gelu

    def _gelu_python(self, input: Tensor) -> Tensor:
        return input * 0.5 * (1.0 + torch.erf(input / math.sqrt(2.0)))

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**GELUActivation Class Summary**

**Purpose and Functionality:**
The `GELUActivation` class implements the Generalized Erifunctional Unit (GELU) activation function. It provides two different implementations: one written in Python (`_gelu_python`) for flexibility, and another using the native PyTorch implementation (`nn.functional.gelu`). The primary purpose is to apply the GELU activation function to input tensors.

**Key Implementation Details:**

* Uses `torch.erf` to compute the error function.
* Employs a hybrid approach with two implementable methods (_gelu_python and the native `nn.functional.gelu`).
 
**Notable Dependencies/Requirements:**
- `matplotlib` (not explicitly used but referenced in related math functions)
- `math` for mathematical operations like sqrt
- `torch.nn.functional` for native implementation


key: chroma:document
string_value:
CODE:

class FastGELUActivation(nn.Module):
    """
    Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs
    """

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**FastGELUActivation Class Summary**

### Purpose and Functionality
This class implements the FastGELU (Gaussian Error Linear Units) activation function, which approximates the standard GELU but is less computationally intensive. It is designed to provide accurate results.

### Key Implementation Details

* The `forward` method takes an input tensor and returns a tensor with the output of the GELU approximation.
* The formula used is: `0.5 * input * (1.0 + torch.tanh(input * 0.7978845608 * (1.0 + 0.044715 * input * input)))`

### Notable Dependencies and Requirements

* `torch`: a PyTorch module required for numerical computations.
* `nn.Module`: the base class for PyTorch neural network modules, inherited by this class.

### Interface
`fastgelu_activation(nn.Module)`

### Return Types

`Tensor`
```


key: chroma:document
string_value:
CODE:

class QuickGELUActivation(nn.Module):
    """
    Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs
    """

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**Summary**

The `QuickGELUActivation` class implements a fast but inaccurate GELU (Gaussian Error Linear Unit) activation function using the `torch` library.

* Primary purpose: Approximates the GELU function for faster computations.
* Key implementation details:
	+ Uses the Gaussian error linear unit function, approximately implemented as `x * sigmoid(1.702 * x)`.

**Code Analysis**

* The class inherits from `nn.Module`, indicating it is a PyTorch neural network module.
* The `forward` method takes an input tensor and returns the output of the approximation, utilizing the `torch` library for the sigmoid function compute.
* No external dependencies are listed in the code snippet. However, it relies on the `torch` library for its implementation.

**Notable Dependencies**

None explicitly listed; relies on `torch` as primary dependency.


key: chroma:document
string_value:
CODE:

class ClippedGELUActivation(nn.Module):
    """
    Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as
    it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to
    https://arxiv.org/abs/2004.09602.

    Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when
    initially created.

    For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 +
    torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://arxiv.org/abs/1606.08415
    """

    def __init__(self, min: float, max: float):
        if min > max:
            raise ValueError(f"min should be < max (got min: {min}, max: {max})")

        super().__init__()
        self.min = min
        self.max = max

    def forward(self, x: Tensor) -> Tensor:


SUMMARY:
**Class Summary: ClippedGELUActivation**

The `ClippedGELUActivation` class is a custom neural network module that applies the clipped GELU (Gaussian Error Linear Unit) activation function. It reduces the range of possible GELu outputs between `[min, max]` to make the activation function more suitable for quantization.

**Key Implementation Details:**

* The `forward` method takes input `x` and returns the clipped GE LU output using `torch.clip`.
* The `__init__` method initializes the module with `min` and `max` values, raising an error if `min` is greater than `max`.

**Notable Dependencies:**

* `gelu`: The original implementation of the GE LU activation function.
* `torch`: Used for mathematical operations.

**Interfaces Implemented:**

* `nn.Module`
* `TensorProcessing` (via `torch.clip`)
* `ActivationFunction` (with custom clipped implementation)


key: chroma:document
string_value:
CODE:

class AccurateGELUActivation(nn.Module):
    """
    Applies GELU approximation that is faster than default and more accurate than QuickGELU. See:
    https://github.com/hendrycks/GELUs

    Implemented along with MEGA (Moving Average Equipped Gated Attention)
    """

    def __init__(self):
        super().__init__()
        self.precomputed_constant = math.sqrt(2 / math.pi)

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**AccurateGELUActivation Class Summary**

**Main Purpose and Functionality:**

The `AccurateGELUActivation` class implements an Accurate GELU (Gaussian Error Linear Unit) activation function using the precomputed constant from Hendryck's implementation. It is optimized for speed and accuracy compared to default GELU and QuickGELU.

**Key Implementation Details:**

* The activation function is computed using `torch.tanh`, with a custom precomputed constant (`self.precomputed_constant`) to improve performance.
* The input tensor is scaled by a factor (`0.5`) before applying the GELU transformation.

**Notable Dependencies or Requirements:**

* `torch`: PyTorch library, specifically for computational operations such as `math` and `torch.tanh`.
* Relevant classes:
	+ `Tensor`: PyTorch's numerical data structure.
* External dependencies:
	+ Hendryck's GitHub implementation of GELU (`https://github.com/hendrycks/GELUs`).


key: chroma:document
string_value:
CODE:

class MishActivation(nn.Module):
    """
    See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also
    visit the official repository for the paper: https://github.com/digantamisra98/Mish
    """

    def __init__(self):
        super().__init__()
        if version.parse(torch.__version__) < version.parse("1.9.0"):
            self.act = self._mish_python
        else:
            self.act = nn.functional.mish

    def _mish_python(self, input: Tensor) -> Tensor:
        return input * torch.tanh(nn.functional.softplus(input))

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**Concise Summary: Mish Activation Function Implementation**

The `MishActivation` class implements the Mitra et al.'s Mish activation function in PyTorch. Its primary purpose is to provide a self-regularized, non-monotonic activation function.

**Key Features:**

* Conditionally uses either the implementation from the Python repository (`_mish_python`) or PyTorch's built-in `nn.functional.mish`.
* `_mish_python` function applies `torch.tanh` and `nn.functional.softplus` to a given input.

**Notable Dependencies:**

* PyTorch library (version-dependent)
* Official Mish repository (optional, used in Python implementation)

**Important Class/Function Signatures:**

* `forward(input: Tensor) -> Tensor`: Takes an input tensor and applies the activation function.
* `_mish_python(input: Tensor) -> Tensor`: Applies the Mish activation function using PyTorch's softplus and tanh for computation.


key: chroma:document
string_value:
CODE:

class LinearActivation(nn.Module):
    """
    Applies the linear activation function, i.e. forwarding input directly to output.
    """

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**Class Summary**

`LinearActivation`

A neural network module that applies the linear activation function by passing the input tensor directly to the output.

### Key Implementation Details

- **Forward Pass**: The `forward` method takes an input tensor and returns it unchanged, effectively applying no transformation.
  - Returns: `Tensor`
  - Method Signature: `input (Tensor) -> Tensor`

### Notable Dependencies or Requirements

- **Tensor**: Input and output tensors using the PyTorch library (`torch.Tensor`).


key: chroma:document
string_value:
CODE:

class LaplaceActivation(nn.Module):
    """
    Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See
    https://arxiv.org/abs/2209.10655

    Inspired by squared relu, but with bounded range and gradient for better stability
    """

    def forward(self, input, mu=0.707107, sigma=0.282095):
        input = (input - mu).div(sigma * math.sqrt(2.0))


SUMMARY:
**LaplaceActivation Class Summary**

The `LaplaceActivation` class is an implementation of the Laplace activation function, introduced as an attention activation in the MEGA paper.

**Primary Purpose:**
The primary purpose of this class is to apply element-wise activation based on the Laplace function.

**Key Implementation Details:**

* The `forward` method calculates the input using the following formula:
	+ `[input - mu] / (sigma * sqrt(2.0))`: standardization around mean (`mu`) and standard deviation (`sigma`)
	+ `torch.erf(input)`: computes the error function of the standardized input
	+ `0.5 * (1.0 + torch.erf(input))`: applies activation to produce output

**Notable Dependencies:**

* Reliably referenced classes and functions:
	+ `torch.erf()`, `math.sqrt()` for computing error function and square root respectively.
* External dependencies:
	+ None explicitly listed, but it is assumed that the PyTorch library (`nn.Module`) is present.

Note this code appears to be incomplete in its implementation as there's no return statement declared outside of the return statement of `torch.erf(input)*0.5`


key: chroma:document
string_value:
CODE:

class ReLUSquaredActivation(nn.Module):
    """
    Applies the relu^2 activation introduced in https://arxiv.org/abs/2109.08668v2
    """

    def forward(self, input):
        relu_applied = nn.functional.relu(input)
        squared = torch.square(relu_applied)


SUMMARY:
**Summary**

`ReLUSquaredActivation` class applies the **ReLU^2** activation function to input data. The primary purpose is to map negative values to zero, similar to ReLU.

**Key Implementation Details:**

* Applies the `ReLU` (Rectified Linear Unit) activation function via `nn.functional.relu(input)`
* Computes the square of the result using `torch.square(relu_applied)`

**Notable Dependencies/Requirements:**

* `nn.Module`
* `nn.functional.ReLU`
* `torch.square`


key: chroma:document
string_value:
CODE:

class ClassInstantier(OrderedDict):
    def __getitem__(self, key):
        content = super().__getitem__(key)
        cls, kwargs = content if isinstance(content, tuple) else (content, {})


SUMMARY:
**Summary:**

ClassInstantier is an implementation of OrderedDict that instantiates class names with keyword arguments when retrieved.

**Key Implementation Details:**

* The `__getitem__` method overrides the base dictionary behavior, utilizing the `super().__getitem__(key)` call to retrieve data.
* It checks if the content is a tuple (i.e., `content = super().__getitem__(key)`) and separates it into class name (`cls`) and keyword arguments (`kwargs`).
* If the content is not a tuple, it assigns the content as the class name and an empty dictionary as keyword arguments.

**Notable Dependencies and Requirements:**

* References:
	+ `super()`
* External Dependencies:
	+ Python's built-in `OrderedDict`
* Inherited or Implemented Interfaces:
	+ None


key: chroma:document
string_value:
CODE:

def get_activation(activation_string):
    if activation_string in ACT2FN:
        return ACT2FN[activation_string]
    else:


SUMMARY:
**Summary**

The `get_activation` function retrieves the activation function corresponding to a given string from the `ACT2FN` dictionary. If the input string is not found, it raises a `KeyError`. The primary purpose of this function is to map string representations of activation functions to their actual functions.

**Implementation Details**

- Key algorithm: Dictionary lookup
- Activation function mapping stored in `ACT2FN` dictionary

**Notable Dependencies/Requirements**

- `ACT2FN`: dictionary containing the mapping between activation function strings and their corresponding functions, e.g., `{ 'relu': npRectify }`.


key: chroma:document
string_value:
CODE:

def _gelu(x):
    """
    Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when
    initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
    0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see
    https://arxiv.org/abs/1606.08415
    """
    x = tf.convert_to_tensor(x)
    cdf = 0.5 * (1.0 + tf.math.erf(x / tf.cast(tf.sqrt(2.0), x.dtype)))



SUMMARY:
**genuine gELU Activation Function**

The `_gelu` function implements the Gaussian Error Linear Unit activation function, initially created in Google's BERT repo.

**Main Purpose and Functionality**

* Applies the gELU activation function to input `x`
* Returns the result of multiplying `x` with the cumulative distribution function (CDF) of the error function (erf)

**Key Implementation Details**

* Uses `tf.math.erf` to compute the inverse error function
* Utilizes numerical stabilization through adding 0.044715 × x³ (OpenAI GPT variant)

**Notable Dependencies and Requirements**

* `tf.convert_to_tensor(x)` converts input to TensorFlow tensor
* `math` module for mathematical functions (specifically, `sqrt` and `pi`)
* `torch` library for PyTorch functionality (`tanh`, `pow`)


key: chroma:document
string_value:
CODE:

def _gelu_new(x):
    """
    Gaussian Error Linear Unit. This is a smoother version of the GELU. Original paper: https://arxiv.org/abs/1606.0841

    Args:
        x: float Tensor to perform activation

    Returns:
        `x` with the GELU activation applied.
    """
    x = tf.convert_to_tensor(x)
    pi = tf.cast(math.pi, x.dtype)
    coeff = tf.cast(0.044715, x.dtype)
    cdf = 0.5 * (1.0 + tf.tanh(tf.sqrt(2.0 / pi) * (x + coeff * tf.pow(x, 3))))



SUMMARY:
Here's a concise RAG system summary for the provided function code chunk:

**Function Purpose:** Applies GELU activation to input tensor `x`.

**Implementation Details:**

- Uses the formula `0.5 * (1.0 + tanh(sqrt(2/pi) * (x+coeff*x^3)))` to compute the activated value.
- Utilizes `tf.tanh` for hyperbolic tangent calculation and `tf.pow` for exponentiation.

**Interfaces and Dependencies:**

* Uses tf module's `tf.convert_to_tensor`, `tf.cast`
* Importantly relies on math.pi constant from Python's built-in math library. 

Note that no external dependencies are mentioned in the given code, implying this function does not introduce any additional external modules.


key: chroma:document
string_value:
CODE:

def mish(x):
    x = tf.convert_to_tensor(x)



SUMMARY:
**Function Summary: `mish`**

The `mish` function applies the Mish activation function to the input tensor `x`. 

* Main purpose: Defines a variant of the Mish activation function, which is defined as `g(x) = x * tf.tanh(tf.math.softplus(x))`, where `tf.math.softplus(x)` is the soft-plus function applied element-wise to `x`.
* Key implementation detail: Applies the `softplus` function from TensorFlow to `x` and then multiplies the result by the hyperbolic tangent of the outcome.
* Notable dependencies:
	+ `tf` (TensorFlow)
	+ `tf.convert_to_tensor`, `tf.tanh`, `tf.math.softplus`


key: chroma:document
string_value:
CODE:

def gelu_fast(x):
    x = tf.convert_to_tensor(x)
    coeff1 = tf.cast(0.044715, x.dtype)
    coeff2 = tf.cast(0.7978845608, x.dtype)



SUMMARY:
**Function Summary: `gelu_fast`**

The `gelu_fast` function implements a simplified version of the GELU (Gaussian Error Linear Units) activation function, designed for efficient computation. The main purpose is to replace the vanilla TensorFlow implementation with a faster variant.

**Key Implementation Details:**

* Multiplies inputs by coefficients (`coeff1` and `coeff2`) for hyperbolic tangent calculation.
* Employs the `tanh` function from TensorFlow for element-wise inversion of the sigmoid function.

**Notable Dependencies/Requirements:**

* `tf` (TensorFlow): used for tensor conversions, casting, and the `tanh` activation function.


key: chroma:document
string_value:
CODE:

def quick_gelu(x):
    x = tf.convert_to_tensor(x)
    coeff = tf.cast(1.702, x.dtype)


SUMMARY:
**Summary:**

The `quick_gelu` function implements a simplified version of the GELU (Gaussian Error Linear Units) activation function. It takes a tensor input `x`, converts it to a TensorFlow tensor, scales coefficients using `tf.math.sigmoid`, and applies element-wise multiplication.

**Implementation Details:**

* Applies scaling factor `coeff` from 1.702.
* Uses `tf.math.sigmoid` to scale input coefficient.
* Multiplies scaled inputs element-wise using binary pointwise operation (`x * sigmoid_coefficient * x`).

**Notable Dependencies:**

* TensorFlow (TF) library, specifically `tf.convert_to_tensor`, `tf.cast`, and `tf.math.sigmoid`.


key: chroma:document
string_value:
CODE:

def gelu_10(x):
    """
    Clip the range of possible GeLU outputs between [-10, 10]. This is especially useful for quantization purpose, as
    it allows mapping 2 negatives values in the GeLU spectrum. For more information on this trick, please refer to
    https://arxiv.org/abs/2004.09602

    Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when
    initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
    0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see
    https://arxiv.org/abs/1606.08415 :param x: :return:
    """


SUMMARY:
**Summary**

`gelu_10(x)` is an implementation of the Gaussian Error Linear Unit (GELU) activation function. It:

* **Crops output range**: Clips GeLU outputs between [-10, 10] for quantization purposes.
* **Original implementation**: Based on an original implementation in Google's BERT repo.
* **Key details**:
	+ Uses `tf.clip_by_value` to clip the output of `_gelu(x)`.
	+ Returns a modified GeLU activation function value.

**Notable dependencies and requirements**

* TensorFlow (`tf`)
* Related class/function: `_gelu(x)`
* External dependency: TensorFlow (TF)

Note: The `_gelu(x)` function is not shown in this code snippet, but it's likely a previously implemented implementation of the GELU activation function.


key: chroma:document
string_value:
CODE:

def glu(x, axis=-1):
    """
    Gated Linear Unit. Implementation as defined in the original paper (see https://arxiv.org/abs/1612.08083), where
    the input `x` is split in two halves across a dimension (`axis`), A and B, returning A * sigmoid(B).

    Args:
        `x`: float Tensor to perform activation
        `axis`: dimension across which `x` be split in half

    Returns:
        `x` with the GLU activation applied (with its size halved across the dimension `axis`).
    """
    a, b = tf.split(x, 2, axis=axis)


SUMMARY:
Here is a concise summary of the provided code chunk:

**Function Summary:**

The `glu` function applies the Gated Linear Unit (GLU) activation to a given float Tensor `x`. It splits `x` into two halves along a specified dimension (`axis`) using `tf.split`, then multiplies the first half with the sigmoid of the second half.

**Key Details:**

* Splits input tensor `x` in half across dimension `axis` using `tf.split`.
* Applies GLU activation by multiplying the first half (`a`) with the sigmoid of the second half (`b`) (`a * tf.math.sigmoid(b)`).
* Returns the modified tensor with its size halved along axis `axis`.

**Notable Dependencies/Requirements:**

* TensorFlow library (`tf` namespace)
* Sigmoid function from `tf.math` module
* Integer division not explicitly demonstrated, but assumed as part of tensor splitting functionality.


key: chroma:document
string_value:
CODE:

def get_tf_activation(activation_string):
    if activation_string in ACT2FN:
        return ACT2FN[activation_string]
    else:


SUMMARY:
**Summary:**

This function takes an `activation_string` as input and returns the corresponding activation function from the `ACT2FN` dictionary. If the string is not found in the dictionary, it raises a KeyError with information about the missing function.

**Key Implementation Details:**

* Uses a dictionary lookup (`ACT2FN[activation_string]`) to retrieve the activation function.
* Raises a KeyError if the activation string is not found in the dictionary.

**Notable Dependencies and Requirements:**

* None

Note: The code assumes that `ACT2FN` is an existing dictionary predefined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def _get_unpad_data(attention_mask: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, int]:
    """
    Retrieves indexing data required to repad unpadded (ragged) tensors.

    Arguments:
        attention_mask (`torch.Tensor`):
            Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid.

    Return:
        indices (`torch.Tensor`):
            The indices of non-masked tokens from the flattened input sequence.
        cu_seqlens (`torch.Tensor`):
            The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,).
        max_seqlen_in_batch (`int`):
            Maximum sequence length in batch.
    """
    seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
    indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
    max_seqlen_in_batch = seqlens_in_batch.max().item()
    cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.int32), (1, 0))
    return (
        indices,
        cu_seqlens,
        max_seqlen_in_batch,


SUMMARY:
**Summary:**

 `_get_unpad_data` calculates indexing data for repadding unpadded tensors.

This function takes an `attention_mask` tensor as input and returns three outputs:

1. **indices**: non-masked token indices from the flattened input sequence (Shape: `(num_samples,)`)
2. **cu_seqlens**: cumulative sequence lengths with padded boundaries (Shape: `(batch_size + 1,)`)
3. **max_seqlen_in_batch**: maximum sequence length in batch (Type: `int`)

**Implementation Details:**

* Calculates sequence lengths using `attention_mask.sum(dim=-1)`
* Extracts non-masked token indices using `torch.nonzero`
* Computes cumulative sequence lengths with padded boundaries using `F.pad` and `torch.cumsum`
* Returns the calculated outputs

**Notable Dependencies/Requirements:**

* `torch.Tensor`
* `torch.noise`
* `torch.nn.functional`, specifically `F.pad`


key: chroma:document
string_value:
CODE:

def _upad_input(
    query_layer: torch.Tensor,
    key_layer: torch.Tensor,
    value_layer: torch.Tensor,
    attention_mask: torch.Tensor,
    query_length: int,
):
    """
    Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches.

    This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary
    tensors for query, key, value tensors.

    Arguments:
        query_layer (`torch.Tensor`):
            Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim).
        key_layer (`torch.Tensor`):
            Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
        value_layer (`torch.Tensor`):
            Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
        attention_mask (`torch.Tensor`):
            Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid.
        query_length (`int`):
            Target length.

    Return:
        query_layer (`torch.Tensor`):
            Query state without padding. Shape: (total_target_length, num_heads, head_dim).
        key_layer (`torch.Tensor`):
            Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
        value_layer (`torch.Tensor`):
            Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
        indices_q (`torch.Tensor`):
            The indices of non-masked tokens from the flattened input target sequence.
        (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`):
            The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,).
        (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`):
            Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value).
    """
    indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(attention_mask)
    batch_size, kv_seq_len, num_key_value_heads, head_dim = key_layer.shape

    key_layer = index_first_axis(key_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k)
    value_layer = index_first_axis(
        value_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
    )
    if query_length == kv_seq_len:
        query_layer = index_first_axis(query_layer.reshape(batch_size * kv_seq_len, -1, head_dim), indices_k)
        cu_seqlens_q = cu_seqlens_k
        max_seqlen_in_batch_q = max_seqlen_in_batch_k
        indices_q = indices_k
    elif query_length == 1:
        max_seqlen_in_batch_q = 1
        cu_seqlens_q = torch.arange(
            batch_size + 1, dtype=torch.int32, device=query_layer.device
        )  # There is a memcpy here, that is very bad.
        indices_q = cu_seqlens_q[:-1]
        query_layer = query_layer.squeeze(1)
    else:
        # The -q_len: slice assumes left padding.
        attention_mask = attention_mask[:, -query_length:]
        query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(query_layer, attention_mask)

    return (
        query_layer,
        key_layer,
        value_layer,
        indices_q,
        (cu_seqlens_q, cu_seqlens_k),
        (max_seqlen_in_batch_q, max_seqlen_in_batch_k),


SUMMARY:
**Summary**

This function, `_upad_input`, is used to unpadd query, key, and value tensors for a single attention operation. It takes in four input tensors: `query_layer`, `key_layer`, `value_layer`, and `attention_mask`. The function returns the unpadded tensors along with additional metadata such as sequence lengths and indices.

**Key Features**

* Unpads query, key, and value tensors using a single dimension for all tokens.
* Avoids recomputing intermediary tensors for query, key, and value using flash_attn.bert_padding.unpad_input.
* Handles case where query length is equal to the number of key/value sequences (KV) or 1.

**Implementation Details**

* Uses `_get_unpad_data` function to extract sequence lengths and indices from `attention_mask`.
* Reshapes tensors to flatten them along the batch and KV axis.
* Applies indexing using `indices_k` to move padding tokens out of the way.
* Handles cases where query length is 1 by applying squeeze operation.

**Notable Dependencies/Requirements**

* `torch.Tensor`
* _get_unpad_data function (not shown in this snippet)
* flash_attn.bert_padding.unpad_input (used for comparison)


key: chroma:document
string_value:
CODE:

def prepare_fa2_from_position_ids(query, key, value, position_ids):
    """
    This function returns necessary arguments to call `flash_attn_varlen_func`.
    All three query, key, value states will be flattened.
    Cummulative lengths of each examples in the batch will be extracted from position_ids.

    NOTE: ideally cummulative lengths should be prepared at the data collator stage

    Arguments:
        query (`torch.Tensor`):
            Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim).
        key (`torch.Tensor`):
            Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
        value (`torch.Tensor`):
            Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
        position_ids (`torch.Tensor`):
            Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid.

    Return:
        query (`torch.Tensor`):
            Query state without padding. Shape: (total_target_length, num_heads, head_dim).
        key (`torch.Tensor`):
            Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
        value (`torch.Tensor`):
            Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
        indices_q (`torch.Tensor`):
            The indices of non-masked tokens from the flattened input target sequence.
        (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`):
            The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,).
        (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`):
            Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value).
    """
    query = query.view(-1, query.size(-2), query.size(-1))
    key = key.contiguous().view(-1, key.size(-2), key.size(-1))
    value = value.contiguous().view(-1, value.size(-2), value.size(-1))
    position_ids = position_ids.flatten()
    indices_q = torch.arange(position_ids.size(0), device=position_ids.device, dtype=torch.int32)

    cu_seq_lens = torch.cat(
        (
            indices_q[position_ids == 0],
            torch.tensor(position_ids.size(), device=position_ids.device, dtype=torch.int32),
        )
    )

    max_length = position_ids.max() + 1



SUMMARY:
**Summary:**

This function `prepare_fa2_from_position_ids` prepares necessary arguments for the `flash_attn_varlen_func` by flattening input tensors and extracting cumulative sequence lengths from `position_ids`.

**Key Details:**

* Flattens query, key, value tensors with padding.
* Extracts indices of non-masked tokens from query using `torch.arange`.
* Computes cumulative sequence lengths (cu_seq_lens) for both target (query) and source (key, value).
* Calculates maximum sequence length in batch (`max_seqlen_in_batch_q` for query, `max_seqlen_in_batch_k` for key/value).

**Return Values:**

* Unpadded query tensor (`torch.Tensor`)
* Unpadded key tensor (`torch.Tensor`)
* Unpadded value tensor (`torch.Tensor`)
* Indices of non-masked tokens in query tensor (`torch.Tensor`)
* Cumulative sequence lengths tuple (`(cu_seq_lens, cu_seq_lens)`)
* Maximum sequence length tuples (`(max_seqlen_in_batch_q, max_seqlen_in_batch_k)`)


key: chroma:document
string_value:
CODE:

def fa_peft_integration_check(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    target_dtype: Optional[torch.dtype] = None,
):
    """
    PEFT usually casts the layer norms in float32 for training stability reasons
    therefore the input hidden states gets silently casted in float32. Hence, we need
    cast them back in float16 / bfloat16 just to be sure everything works as expected.
    This might slowdown training & inference so it is recommended to not cast the LayerNorms!

    Args:
        query (`torch.Tensor`):
            Input query states to be passed to Flash Attention API
        key (`torch.Tensor`):
            Input key states to be passed to Flash Attention API
        value (`torch.Tensor`):
            Input value states to be passed to Flash Attention API
        target_dtype (`torch.dtype`, *optional*):
            The dtype to convert the attention tensors to. Conversion can be ignored by
            not providing the target dtype.
    """
    if target_dtype is None:
        return query, key, value

    input_dtype = value.dtype
    if input_dtype == torch.float32:
        logger.warning_once(
            f"The input hidden states seems to be silently casted in float32, this might be related to"
            f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in"
            f" {target_dtype}."
        )

        query = query.to(target_dtype)
        key = key.to(target_dtype)
        value = value.to(target_dtype)



SUMMARY:
**Function Summary: `fa_peft_integration_check`**

**Purpose:** Casts input tensors (`query`, `key`, `value`) to a specified target data type (`target_dtype`) in the PEFT integration check.

**Key Implementation Details:**

* Casts input tensors to `float16/bfloat16` if input is already in `torch.float32` for stability reasons.
* Supports optional `target_dtype` conversion, which can be ignored for compatibility.
* Uses `logger.warning_once` to log potential issues related to silent casting.

**Method Signatures and Interfaces:**
```python
import torch as torch

def fa_peft_integration_check(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    target_dtype: Optional[torch.dtype] = None,
):
```

**External Dependencies:**

* `torch` for tensor operations, including data type conversions.
* `logger` for logging warnings (not shown in this code snippet).


key: chroma:document
string_value:
CODE:

def _flash_attention_forward(
    query_states: torch.Tensor,
    key_states: torch.Tensor,
    value_states: torch.Tensor,
    attention_mask: torch.Tensor,
    query_length: int,
    is_causal: bool,
    dropout: float = 0.0,
    position_ids: Optional[torch.Tensor] = None,
    softmax_scale: Optional[float] = None,
    sliding_window: Optional[int] = None,
    use_top_left_mask: bool = False,
    softcap: Optional[float] = None,
    deterministic: bool = None,
    cu_seq_lens_q: Optional[torch.LongTensor] = None,
    cu_seq_lens_k: Optional[torch.LongTensor] = None,
    max_length_q: Optional[int] = None,
    max_length_k: Optional[int] = None,
    target_dtype: Optional[torch.dtype] = None,
    **kwargs,
):
    """
    Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token
    first unpad the input, then computes the attention scores and pad the final attention scores.

    Args:
        query_states (`torch.Tensor`):
            Input query states to be passed to Flash Attention API
        key_states (`torch.Tensor`):
            Input key states to be passed to Flash Attention API
        value_states (`torch.Tensor`):
            Input value states to be passed to Flash Attention API
        attention_mask (`torch.Tensor`):
            The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the
            position of padding tokens and 1 for the position of non-padding tokens.
        dropout (`float`):
            Attention dropout
        softmax_scale (`float`, *optional*):
            The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim)
        use_top_left_mask (`bool`, defaults to `False`):
            flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference.
        softcap (`float`, *optional*):
            Softcap for the attention logits, used e.g. in gemma2.
        deterministic (`bool`, *optional*):
            Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled.
    """
    if not use_top_left_mask:
        causal = is_causal
    else:
        # TODO: Remove the `query_length != 1` check once Flash Attention for RoCm is bumped to 2.1.
        causal = is_causal and query_length != 1

    # Assuming 4D tensors, key_states.shape[1] is the key/value sequence length (source length).
    use_sliding_windows = (
        _flash_supports_window_size and sliding_window is not None and key_states.shape[1] > sliding_window
    )
    flash_kwargs = {"window_size": (sliding_window, sliding_window)} if use_sliding_windows else {}

    if flash_241:
        if deterministic is None:
            deterministic = deterministic_g
        flash_kwargs["deterministic"] = deterministic

    if softcap is not None:
        flash_kwargs["softcap"] = softcap

    # PEFT possibly silently casts tensors to fp32, this potentially reconverts to correct dtype or is a no op
    query_states, key_states, value_states = fa_peft_integration_check(
        query_states, key_states, value_states, target_dtype
    )

    # Contains at least one padding token in the sequence
    if attention_mask is not None:
        batch_size = query_states.shape[0]
        query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = _upad_input(
            query_states, key_states, value_states, attention_mask, query_length
        )
        cu_seqlens_q, cu_seqlens_k = cu_seq_lens
        max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens

        attn_output_unpad = flash_attn_varlen_func(
            query_states,
            key_states,
            value_states,
            cu_seqlens_q=cu_seqlens_q,
            cu_seqlens_k=cu_seqlens_k,
            max_seqlen_q=max_seqlen_in_batch_q,
            max_seqlen_k=max_seqlen_in_batch_k,
            dropout_p=dropout,
            softmax_scale=softmax_scale,
            causal=causal,
            **flash_kwargs,
        )
        attn_output = pad_input(attn_output_unpad, indices_q, batch_size, query_length)

    # If position_ids is provided and check all examples do not contain only 1 sequence, If tensor in increasing
    # then we probably have one sequence, otherwise it is packed. Additionally check we are in pre-fill/training stage.
    # Use `flash_attn_varlen_func` to prevent cross-example attention and also allow padding free approach
    elif position_ids is not None and (
        max_length_q is not None or (query_length != 1 and not (torch.diff(position_ids, dim=-1) >= 0).all())
    ):
        batch_size = query_states.size(0)

        if cu_seq_lens_q is None or cu_seq_lens_k is None:
            query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = (
                prepare_fa2_from_position_ids(query_states, key_states, value_states, position_ids)
            )

            cu_seq_lens_q, cu_seq_lens_k = cu_seq_lens
            max_length_q, max_length_k = max_seq_lens

        else:
            query_states = query_states.reshape(-1, query_states.size(-2), query_states.size(-1))
            key_states = key_states.reshape(-1, key_states.size(-2), key_states.size(-1))
            value_states = value_states.reshape(-1, value_states.size(-2), value_states.size(-1))

        attn_output = flash_attn_varlen_func(
            query_states,
            key_states,
            value_states,
            cu_seqlens_q=cu_seq_lens_q,
            cu_seqlens_k=cu_seq_lens_k,
            max_seqlen_q=max_length_q,
            max_seqlen_k=max_length_k,
            dropout_p=dropout,
            softmax_scale=softmax_scale,
            causal=causal,
            **flash_kwargs,
        )

        attn_output = attn_output.view(batch_size, -1, attn_output.size(-2), attn_output.size(-1))

    else:
        attn_output = flash_attn_func(
            query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=causal, **flash_kwargs
        )



SUMMARY:
**Function Summary: `_flash_attention_forward`**

**Purpose and Functionality:**
The `_flash_attention_forward` function implements the forward pass of the Flash Attention mechanism in transformer models. It takes query, key, value states, an attention mask, and optional parameters such as dropout rate, softmax scaling factor, and position IDs.

**Key Implementation Details:**

* The function checks for the presence of padding tokens in the input sequences and un pads them if necessary.
* It uses the `flash_attn` function to compute attention scores and the `pad_input` function to pad the final attention output.
* If position IDs are provided, it uses a variant of the Flash Attention mechanism that allows for packing multiple sequences into a single tensor.

**Notable Dependencies and Requirements:**

* `torch` library for tensor operations
* `flash_attn` function, which is not shown in this code snippet
* `_upad_input`, `fa_peft_integration_check`, and `prepare_fa2_from_position_ids` functions, which are not shown in this code snippet

**Method Signatures and Interfaces:**

* `flash_attn_varlen_func`: computes attention scores with variable-length sequences.
* `pad_input`: pads the final attention output to ensure it has a specific shape.

Note that the exact method signatures and interfaces used by these functions are not explicitly stated in the provided code snippet.


key: chroma:document
string_value:
CODE:

class FlashAttentionKwargs(TypedDict, total=False):
    """
    Keyword arguments for Flash Attention with Compile.

    Attributes:
        cu_seq_lens_q (`torch.LongTensor`, *optional*)
            Gets cumlative sequence length for query state.
        cu_seq_lens_k (`torch.LongTensor`, *optional*)
            Gets cumlative sequence length for key state.
        max_length_q (`int`, *optional*):
            Maximum sequence length for query state.
        max_length_k (`int`, *optional*):
            Maximum sequence length for key state.
    """

    cu_seq_lens_q: Optional[torch.LongTensor]
    cu_seq_lens_k: Optional[torch.LongTensor]
    max_length_q: Optional[int]


SUMMARY:
**Summary**

The `FlashAttentionKwargs` class defines keyword arguments for Flash Attention, a neural network architecture compound (Cumlab). It contains optional attributes:

*   Cumulative sequence lengths for query and key states (`cu_seq_lens_q` and `cu_seq_lens_k`)
*   Maximum allowed sequence lengths for query and key states (`max_length_q` and `max_length_k`)

**Key Implementation Details**

*   Uses `torch(LongTensor)` as data type for cumulative sequence lengths
*   Enforces optional attribute usage (i.e., variables are labeled as *optional*)

**Notable Dependencies or Requirements**

*   Requires the PyTorch library (`torch`)
*   Inherently uses Cumlab's Compile feature (not explicitly mentioned but implied by class name and purpose)


key: docstring
string_value:
# TODO: Move BatchFeature to be imported by both image_processing_utils and image_processing_utils
# We override the class string here, but logic is the same.


key: chroma:document
string_value:
CODE:
# We override the class string here, but logic is the same.
class BatchFeature(BaseBatchFeature):
    r"""
    Holds the output of the image processor specific `__call__` methods.

    This class is derived from a python dictionary and can be used as a dictionary.

    Args:
        data (`dict`):
            Dictionary of lists/arrays/tensors returned by the __call__ method ('pixel_values', etc.).
        tensor_type (`Union[None, str, TensorType]`, *optional*):
            You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at
            initialization.


SUMMARY:
**Summary:**

`BatchFeature` class:

- Derived from `BaseBatchFeature` (class inherited).
- Represents output of image processor's `__call__` methods as a dictionary.
- Accepts `data` (dictionary) and optional `tensor_type` for tensor conversion during initialization.

**Key Implementation Detail:**
The key implementation detail is the override of the class string, but actual logic remains in the parent class (`BaseBatchFeature`).

**Notable Dependency/Requirement:**
- `BaseBatchFeature`: Inherited from.
- `Union[None, str, TensorType]`: Optional tensor type requirement.

**External Dependencies:**

- `image_processing_utils`: Import module for the class.


key: chroma:document
string_value:
CODE:
# TODO: (Amy) - factor out the common parts of this and the feature extractor
class ImageProcessingMixin(PushToHubMixin):
    """
    This is an image processor mixin used to provide saving/loading functionality for sequential and image feature
    extractors.
    """

    _auto_class = None

    def __init__(self, **kwargs):
        """Set elements of `kwargs` as attributes."""
        # This key was saved while we still used `XXXFeatureExtractor` for image processing. Now we use
        # `XXXImageProcessor`, this attribute and its value are misleading.
        kwargs.pop("feature_extractor_type", None)
        # Pop "processor_class" as it should be saved as private attribute
        self._processor_class = kwargs.pop("processor_class", None)
        # Additional attributes without default values
        for key, value in kwargs.items():
            try:
                setattr(self, key, value)
            except AttributeError as err:
                logger.error(f"Can't set {key} with value {value} for {self}")
                raise err

    def _set_processor_class(self, processor_class: str):
        """Sets processor class as an attribute."""
        self._processor_class = processor_class

    @classmethod
    def from_pretrained(
        cls: Type[ImageProcessorType],
        pretrained_model_name_or_path: Union[str, os.PathLike],
        cache_dir: Optional[Union[str, os.PathLike]] = None,
        force_download: bool = False,
        local_files_only: bool = False,
        token: Optional[Union[str, bool]] = None,
        revision: str = "main",
        **kwargs,
    ) -> ImageProcessorType:
        r"""
        Instantiate a type of [`~image_processing_utils.ImageProcessingMixin`] from an image processor.

        Args:
            pretrained_model_name_or_path (`str` or `os.PathLike`):
                This can be either:

                - a string, the *model id* of a pretrained image_processor hosted inside a model repo on
                  huggingface.co.
                - a path to a *directory* containing a image processor file saved using the
                  [`~image_processing_utils.ImageProcessingMixin.save_pretrained`] method, e.g.,
                  `./my_model_directory/`.
                - a path or url to a saved image processor JSON *file*, e.g.,
                  `./my_model_directory/preprocessor_config.json`.
            cache_dir (`str` or `os.PathLike`, *optional*):
                Path to a directory in which a downloaded pretrained model image processor should be cached if the
                standard cache should not be used.
            force_download (`bool`, *optional*, defaults to `False`):
                Whether or not to force to (re-)download the image processor files and override the cached versions if
                they exist.
            resume_download:
                Deprecated and ignored. All downloads are now resumed by default when possible.
                Will be removed in v5 of Transformers.
            proxies (`Dict[str, str]`, *optional*):
                A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
                'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
            token (`str` or `bool`, *optional*):
                The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use
                the token generated when running `huggingface-cli login` (stored in `~/.huggingface`).
            revision (`str`, *optional*, defaults to `"main"`):
                The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
                git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
                identifier allowed by git.


                <Tip>

                To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>"`.

                </Tip>

            return_unused_kwargs (`bool`, *optional*, defaults to `False`):
                If `False`, then this function returns just the final image processor object. If `True`, then this
                functions returns a `Tuple(image_processor, unused_kwargs)` where *unused_kwargs* is a dictionary
                consisting of the key/value pairs whose keys are not image processor attributes: i.e., the part of
                `kwargs` which has not been used to update `image_processor` and is otherwise ignored.
            subfolder (`str`, *optional*, defaults to `""`):
                In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
                specify the folder name here.
            kwargs (`Dict[str, Any]`, *optional*):
                The values in kwargs of any keys which are image processor attributes will be used to override the
                loaded values. Behavior concerning key/value pairs whose keys are *not* image processor attributes is
                controlled by the `return_unused_kwargs` keyword parameter.

        Returns:
            A image processor of type [`~image_processing_utils.ImageProcessingMixin`].

        Examples:

        ```python
        # We can't instantiate directly the base class *ImageProcessingMixin* so let's show the examples on a
        # derived class: *CLIPImageProcessor*
        image_processor = CLIPImageProcessor.from_pretrained(
            "openai/clip-vit-base-patch32"
        )  # Download image_processing_config from huggingface.co and cache.
        image_processor = CLIPImageProcessor.from_pretrained(
            "./test/saved_model/"
        )  # E.g. image processor (or model) was saved using *save_pretrained('./test/saved_model/')*
        image_processor = CLIPImageProcessor.from_pretrained("./test/saved_model/preprocessor_config.json")
        image_processor = CLIPImageProcessor.from_pretrained(
            "openai/clip-vit-base-patch32", do_normalize=False, foo=False
        )
        assert image_processor.do_normalize is False
        image_processor, unused_kwargs = CLIPImageProcessor.from_pretrained(
            "openai/clip-vit-base-patch32", do_normalize=False, foo=False, return_unused_kwargs=True
        )
        assert image_processor.do_normalize is False
        assert unused_kwargs == {"foo": False}
        ```"""
        kwargs["cache_dir"] = cache_dir
        kwargs["force_download"] = force_download
        kwargs["local_files_only"] = local_files_only
        kwargs["revision"] = revision

        use_auth_token = kwargs.pop("use_auth_token", None)
        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token

        if token is not None:
            kwargs["token"] = token

        image_processor_dict, kwargs = cls.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)

        return cls.from_dict(image_processor_dict, **kwargs)

    def save_pretrained(self, save_directory: Union[str, os.PathLike], push_to_hub: bool = False, **kwargs):
        """
        Save an image processor object to the directory `save_directory`, so that it can be re-loaded using the
        [`~image_processing_utils.ImageProcessingMixin.from_pretrained`] class method.

        Args:
            save_directory (`str` or `os.PathLike`):
                Directory where the image processor JSON file will be saved (will be created if it does not exist).
            push_to_hub (`bool`, *optional*, defaults to `False`):
                Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
                repository you want to push to with `repo_id` (will default to the name of `save_directory` in your
                namespace).
            kwargs (`Dict[str, Any]`, *optional*):
                Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.
        """
        use_auth_token = kwargs.pop("use_auth_token", None)

        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if kwargs.get("token", None) is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            kwargs["token"] = use_auth_token

        if os.path.isfile(save_directory):
            raise AssertionError(f"Provided path ({save_directory}) should be a directory, not a file")

        os.makedirs(save_directory, exist_ok=True)

        if push_to_hub:
            commit_message = kwargs.pop("commit_message", None)
            repo_id = kwargs.pop("repo_id", save_directory.split(os.path.sep)[-1])
            repo_id = self._create_repo(repo_id, **kwargs)
            files_timestamps = self._get_files_timestamps(save_directory)

        # If we have a custom config, we copy the file defining it in the folder and set the attributes so it can be
        # loaded from the Hub.
        if self._auto_class is not None:
            custom_object_save(self, save_directory, config=self)

        # If we save using the predefined names, we can load using `from_pretrained`
        output_image_processor_file = os.path.join(save_directory, IMAGE_PROCESSOR_NAME)

        self.to_json_file(output_image_processor_file)
        logger.info(f"Image processor saved in {output_image_processor_file}")

        if push_to_hub:
            self._upload_modified_files(
                save_directory,
                repo_id,
                files_timestamps,
                commit_message=commit_message,
                token=kwargs.get("token"),
            )

        return [output_image_processor_file]

    @classmethod
    def get_image_processor_dict(
        cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs
    ) -> Tuple[Dict[str, Any], Dict[str, Any]]:
        """
        From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a
        image processor of type [`~image_processor_utils.ImageProcessingMixin`] using `from_dict`.

        Parameters:
            pretrained_model_name_or_path (`str` or `os.PathLike`):
                The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
            subfolder (`str`, *optional*, defaults to `""`):
                In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
                specify the folder name here.
            image_processor_filename (`str`, *optional*, defaults to `"config.json"`):
                The name of the file in the model directory to use for the image processor config.

        Returns:
            `Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the image processor object.
        """
        cache_dir = kwargs.pop("cache_dir", None)
        force_download = kwargs.pop("force_download", False)
        resume_download = kwargs.pop("resume_download", None)
        proxies = kwargs.pop("proxies", None)
        token = kwargs.pop("token", None)
        use_auth_token = kwargs.pop("use_auth_token", None)
        local_files_only = kwargs.pop("local_files_only", False)
        revision = kwargs.pop("revision", None)
        subfolder = kwargs.pop("subfolder", "")
        image_processor_filename = kwargs.pop("image_processor_filename", IMAGE_PROCESSOR_NAME)

        from_pipeline = kwargs.pop("_from_pipeline", None)
        from_auto_class = kwargs.pop("_from_auto", False)

        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token

        user_agent = {"file_type": "image processor", "from_auto_class": from_auto_class}
        if from_pipeline is not None:
            user_agent["using_pipeline"] = from_pipeline

        if is_offline_mode() and not local_files_only:
            logger.info("Offline mode: forcing local_files_only=True")
            local_files_only = True

        pretrained_model_name_or_path = str(pretrained_model_name_or_path)
        is_local = os.path.isdir(pretrained_model_name_or_path)
        if os.path.isdir(pretrained_model_name_or_path):
            image_processor_file = os.path.join(pretrained_model_name_or_path, image_processor_filename)
        if os.path.isfile(pretrained_model_name_or_path):
            resolved_image_processor_file = pretrained_model_name_or_path
            is_local = True
        elif is_remote_url(pretrained_model_name_or_path):
            image_processor_file = pretrained_model_name_or_path
            resolved_image_processor_file = download_url(pretrained_model_name_or_path)
        else:
            image_processor_file = image_processor_filename
            try:
                # Load from local folder or from cache or download from model Hub and cache
                resolved_image_processor_file = cached_file(
                    pretrained_model_name_or_path,
                    image_processor_file,
                    cache_dir=cache_dir,
                    force_download=force_download,
                    proxies=proxies,
                    resume_download=resume_download,
                    local_files_only=local_files_only,
                    token=token,
                    user_agent=user_agent,
                    revision=revision,
                    subfolder=subfolder,
                )
            except EnvironmentError:
                # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted to
                # the original exception.
                raise
            except Exception:
                # For any other exception, we throw a generic error.
                raise EnvironmentError(
                    f"Can't load image processor for '{pretrained_model_name_or_path}'. If you were trying to load"
                    " it from 'https://huggingface.co/models', make sure you don't have a local directory with the"
                    f" same name. Otherwise, make sure '{pretrained_model_name_or_path}' is the correct path to a"
                    f" directory containing a {image_processor_filename} file"
                )

        try:
            # Load image_processor dict
            with open(resolved_image_processor_file, "r", encoding="utf-8") as reader:
                text = reader.read()
            image_processor_dict = json.loads(text)

        except json.JSONDecodeError:
            raise EnvironmentError(
                f"It looks like the config file at '{resolved_image_processor_file}' is not a valid JSON file."
            )

        if is_local:
            logger.info(f"loading configuration file {resolved_image_processor_file}")
        else:
            logger.info(
                f"loading configuration file {image_processor_file} from cache at {resolved_image_processor_file}"
            )
            if "auto_map" in image_processor_dict:
                image_processor_dict["auto_map"] = add_model_info_to_auto_map(
                    image_processor_dict["auto_map"], pretrained_model_name_or_path
                )
            if "custom_pipelines" in image_processor_dict:
                image_processor_dict["custom_pipelines"] = add_model_info_to_custom_pipelines(
                    image_processor_dict["custom_pipelines"], pretrained_model_name_or_path
                )

        return image_processor_dict, kwargs

    @classmethod
    def from_dict(cls, image_processor_dict: Dict[str, Any], **kwargs):
        """
        Instantiates a type of [`~image_processing_utils.ImageProcessingMixin`] from a Python dictionary of parameters.

        Args:
            image_processor_dict (`Dict[str, Any]`):
                Dictionary that will be used to instantiate the image processor object. Such a dictionary can be
                retrieved from a pretrained checkpoint by leveraging the
                [`~image_processing_utils.ImageProcessingMixin.to_dict`] method.
            kwargs (`Dict[str, Any]`):
                Additional parameters from which to initialize the image processor object.

        Returns:
            [`~image_processing_utils.ImageProcessingMixin`]: The image processor object instantiated from those
            parameters.
        """
        image_processor_dict = image_processor_dict.copy()
        return_unused_kwargs = kwargs.pop("return_unused_kwargs", False)

        # The `size` parameter is a dict and was previously an int or tuple in feature extractors.
        # We set `size` here directly to the `image_processor_dict` so that it is converted to the appropriate
        # dict within the image processor and isn't overwritten if `size` is passed in as a kwarg.
        if "size" in kwargs and "size" in image_processor_dict:
            image_processor_dict["size"] = kwargs.pop("size")
        if "crop_size" in kwargs and "crop_size" in image_processor_dict:
            image_processor_dict["crop_size"] = kwargs.pop("crop_size")

        image_processor = cls(**image_processor_dict)

        # Update image_processor with kwargs if needed
        to_remove = []
        for key, value in kwargs.items():
            if hasattr(image_processor, key):
                setattr(image_processor, key, value)
                to_remove.append(key)
        for key in to_remove:
            kwargs.pop(key, None)

        logger.info(f"Image processor {image_processor}")
        if return_unused_kwargs:
            return image_processor, kwargs
        else:
            return image_processor

    def to_dict(self) -> Dict[str, Any]:
        """
        Serializes this instance to a Python dictionary.

        Returns:
            `Dict[str, Any]`: Dictionary of all the attributes that make up this image processor instance.
        """
        output = copy.deepcopy(self.__dict__)
        output["image_processor_type"] = self.__class__.__name__

        return output

    @classmethod
    def from_json_file(cls, json_file: Union[str, os.PathLike]):
        """
        Instantiates a image processor of type [`~image_processing_utils.ImageProcessingMixin`] from the path to a JSON
        file of parameters.

        Args:
            json_file (`str` or `os.PathLike`):
                Path to the JSON file containing the parameters.

        Returns:
            A image processor of type [`~image_processing_utils.ImageProcessingMixin`]: The image_processor object
            instantiated from that JSON file.
        """
        with open(json_file, "r", encoding="utf-8") as reader:
            text = reader.read()
        image_processor_dict = json.loads(text)
        return cls(**image_processor_dict)

    def to_json_string(self) -> str:
        """
        Serializes this instance to a JSON string.

        Returns:
            `str`: String containing all the attributes that make up this feature_extractor instance in JSON format.
        """
        dictionary = self.to_dict()

        for key, value in dictionary.items():
            if isinstance(value, np.ndarray):
                dictionary[key] = value.tolist()

        # make sure private name "_processor_class" is correctly
        # saved as "processor_class"
        _processor_class = dictionary.pop("_processor_class", None)
        if _processor_class is not None:
            dictionary["processor_class"] = _processor_class

        return json.dumps(dictionary, indent=2, sort_keys=True) + "\n"

    def to_json_file(self, json_file_path: Union[str, os.PathLike]):
        """
        Save this instance to a JSON file.

        Args:
            json_file_path (`str` or `os.PathLike`):
                Path to the JSON file in which this image_processor instance's parameters will be saved.
        """
        with open(json_file_path, "w", encoding="utf-8") as writer:
            writer.write(self.to_json_string())

    def __repr__(self):
        return f"{self.__class__.__name__} {self.to_json_string()}"

    @classmethod
    def register_for_auto_class(cls, auto_class="AutoImageProcessor"):
        """
        Register this class with a given auto class. This should only be used for custom image processors as the ones
        in the library are already mapped with `AutoImageProcessor `.

        <Tip warning={true}>

        This API is experimental and may have some slight breaking changes in the next releases.

        </Tip>

        Args:
            auto_class (`str` or `type`, *optional*, defaults to `"AutoImageProcessor "`):
                The auto class to register this new image processor with.
        """
        if not isinstance(auto_class, str):
            auto_class = auto_class.__name__

        import transformers.models.auto as auto_module

        if not hasattr(auto_module, auto_class):
            raise ValueError(f"{auto_class} is not a valid auto class.")

        cls._auto_class = auto_class

    def fetch_images(self, image_url_or_urls: Union[str, List[str]]):
        """
        Convert a single or a list of urls into the corresponding `PIL.Image` objects.

        If a single url is passed, the return value will be a single object. If a list is passed a list of objects is
        returned.
        """
        headers = {
            "User-Agent": (
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0"
                " Safari/537.36"
            )
        }
        if isinstance(image_url_or_urls, list):
            return [self.fetch_images(x) for x in image_url_or_urls]
        elif isinstance(image_url_or_urls, str):
            response = requests.get(image_url_or_urls, stream=True, headers=headers)
            response.raise_for_status()
            return Image.open(BytesIO(response.content))
        else:


SUMMARY:
The code provided is for an image processor class in Python, specifically designed to work with the TensorFlow library. It provides methods for processing images, such as resizing, rotating, and flipping, as well as registering itself with a given auto-class.

Here are some key features of this class:

1.  **Initialization**: The class can be initialized with various parameters like image dimensions, cropping, resampling filters etc.
2.  **Image Preprocessing**: It offers methods for preprocessing images such as resizing, rotating, and flipping.
3.  **Registration with Auto-Classes**: This class supports registration with a given auto-class. This is useful when working with custom image processors.

Here is an example of how you might use this class to resize an image by 40%:
```python
from tensorflow.keras.preprocessing.image import load_img, img_to_array

img = load_img("path_to_your_image.jpg", target_size=None)
img = img.resize((int(img.size[0] * 0.4), int(img.size[1] * 0.4)))
img_array = load_img(img, target_size=(224, 224))
print(img_array.shape)
```
This code resizes the image by a factor of 40% (i.e., it crops from the top-right or bottom-left corner 30%) and then converts it to the (224, 224) shape.

**Update:** The following example demonstrates registering this class with an auto-class `AutoImageProcessor` :
```python
from PIL import Image
import requests

class MyImageProcessor(ImageProcessor):
    pass


class AutoImageProcessor:
    def __init__(self):
        pass

# Register the image processor
MyImageProcessor.register_for_auto_class()
```

To use this class effectively with an existing model created in the Hugging Face Transformers library, you may have to first convert the images into NumPy arrays and then load them into the specified dimensions. 

For example, if `model_name` is the name of your pre-trained model:
```python
from tensorflow.keras.applications import ResNet50
from transformers import AutoImageProcessor

# Load my model using AutoImageProcessor
my_model = models.AutoImageProcessor("resnet50")

my_mdl_1d_features = my_model.extract_features([list_to_np(image)], return_tensors="pt", size_divisor=16)
feature_extractor = my_image_processor.to_feature_importance(
    [None, None],
    max_size=3 * 512,
    feature_map_layers=(3,),
    return_as_list=True
)

# Use the model outputs to perform classification or regression...
```


key: chroma:document
string_value:
CODE:

def default_logdir() -> str:
    """
    Same default as PyTorch
    """
    import socket
    from datetime import datetime

    current_time = datetime.now().strftime("%b%d_%H-%M-%S")


SUMMARY:
**Summary:**

This function, `default_logdir()`, generates and returns a default log directory path. It combines the current date-time with a hostname-based suffix and stores it in a "runs" directory.

**Key Details:**

- **Function Signatures:** None
- **Main Purpose:** Generate default log directory path.
- **Key Implementation:** Uses `datetime.now()` for date-time, `socket.gethostname()` for hostname, and `os.path.join()` for directory construction.
- **External Dependencies:** 
  - `socket`
  - `strptime` (from `datetime`)
  - `gethostname` (from `socket`)
  - `os`
- **Interfaces/Inheritance:**
  None


key: chroma:document
string_value:
CODE:

def get_int_from_env(env_keys, default):
    """Returns the first positive env value found in the `env_keys` list or the default."""
    for e in env_keys:
        val = int(os.environ.get(e, -1))
        if val >= 0:
            return val


SUMMARY:
**Code Summary**

This function retrieves an integer value from environment variables specified by `env_keys` and returns the first positive result found. If no positive value is found, it returns the provided `default`.

**Key Implementation Details**

- Uses `os.environ.get()` to retrieve environment variable values.
- Attempts to convert each result to an integer, returning immediately on success.

**Function Signature**

`get_int_from_env(env_keys: list, default: any) -> int`

**Notable Dependencies/Requirements**

- `os`: for accessing environment variables.

No inherited or implemented interfaces noted.


key: chroma:document
string_value:
CODE:

def get_xla_device_type(device: "torch.device") -> Optional[str]:
    """
    Returns the xla device type (CPU|GPU|TPU) or None if the device is a non-xla device.
    """
    if is_torch_xla_available():
        if device.type == "cpu":
            return "CPU"
        return xm.xla_real_devices([device])[0].split(":")[0]


SUMMARY:
**Function Summary**

`get_xla_device_type(device: torch.device) -> Optional[str]`

Returns the XLA device type (CPU, GPU, TPU) or `None` for non-XLA devices.

**Key Features**

* Checks if `torch_xla` is available and returns a valid XLA device type
* Returns "CPU" if the device is a CPU
* Extracts the device type from an XLA real device (GPU/TPU) using `xm.xla_real_devices()`
* Assumes `device.type == ["cpu"]` comparison implicitly matches with string comparison

**Implementation Details**

* Uses `torch.device` object as input
* Utilizes `is_torch_xla_available()` function to check availability
* Utilizes `xm.xla_real_devices()` function to extract XLA device type

**Note**: Assumes familiarity with `torch_xla` and `xm` dependencies. No external dependencies or requirements specified.


key: chroma:document
string_value:
CODE:

class OptimizerNames(ExplicitEnum):
    """
    Stores the acceptable string identifiers for optimizers.
    """

    ADAMW_HF = "adamw_hf"
    ADAMW_TORCH = "adamw_torch"
    ADAMW_TORCH_FUSED = "adamw_torch_fused"
    ADAMW_TORCH_XLA = "adamw_torch_xla"
    ADAMW_TORCH_NPU_FUSED = "adamw_torch_npu_fused"
    ADAMW_APEX_FUSED = "adamw_apex_fused"
    ADAFACTOR = "adafactor"
    ADAMW_ANYPRECISION = "adamw_anyprecision"
    ADAMW_TORCH_4BIT = "adamw_torch_4bit"
    ADEMAMIX = "ademamix"
    SGD = "sgd"
    ADAGRAD = "adagrad"
    ADAMW_BNB = "adamw_bnb_8bit"
    ADAMW_8BIT = "adamw_8bit"  # just an alias for adamw_bnb_8bit
    ADEMAMIX_8BIT = "ademamix_8bit"
    LION_8BIT = "lion_8bit"
    LION = "lion_32bit"
    PAGED_ADAMW = "paged_adamw_32bit"
    PAGED_ADAMW_8BIT = "paged_adamw_8bit"
    PAGED_ADEMAMIX = "paged_ademamix_32bit"
    PAGED_ADEMAMIX_8BIT = "paged_ademamix_8bit"
    PAGED_LION = "paged_lion_32bit"
    PAGED_LION_8BIT = "paged_lion_8bit"
    RMSPROP = "rmsprop"
    RMSPROP_BNB = "rmsprop_bnb"
    RMSPROP_8BIT = "rmsprop_bnb_8bit"
    RMSPROP_32BIT = "rmsprop_bnb_32bit"
    GALORE_ADAMW = "galore_adamw"
    GALORE_ADAMW_8BIT = "galore_adamw_8bit"
    GALORE_ADAFACTOR = "galore_adafactor"
    GALORE_ADAMW_LAYERWISE = "galore_adamw_layerwise"
    GALORE_ADAMW_8BIT_LAYERWISE = "galore_adamw_8bit_layerwise"
    GALORE_ADAFACTOR_LAYERWISE = "galore_adafactor_layerwise"
    LOMO = "lomo"
    ADALOMO = "adalomo"
    GROKADAMW = "grokadamw"
    SCHEDULE_FREE_ADAMW = "schedule_free_adamw"


SUMMARY:
**Summary:**

`OptimizerNames` class defines an enumeration of string identifiers for various optimizer names used in machine learning models. The class is part of the `ExplicitEnum` design pattern.

**Main Purpose and Functionality:**
The class serves as a centralized registry for different optimizer names, making it easier to switch between optimizers or retrieve specific optimizer names throughout the codebase.

**Key Implementation Details:**

* The class contains 42 named constants, each representing an optimizer name (e.g., `ADAMW_HF`, `SGD`, etc.).
* The constants are divided into categories based on their type or characteristics (e.g., mixed precision optimizers like `ABN_BNB_8BIT` and integer-8-bit optimizers like `SGD`, etc.).

**Notable Dependencies or Requirements:**

* No external dependencies are mentioned.
* None of the named constants appear to inherit from a parent class or implement an interface.


key: chroma:document
string_value:
CODE:

def _convert_str_dict(passed_value: dict):
    "Safely checks that a passed value is a dictionary and converts any string values to their appropriate types."
    for key, value in passed_value.items():
        if isinstance(value, dict):
            passed_value[key] = _convert_str_dict(value)
        elif isinstance(value, str):
            # First check for bool and convert
            if value.lower() in ("true", "false"):
                passed_value[key] = value.lower() == "true"
            # Check for digit
            elif value.isdigit():
                passed_value[key] = int(value)
            elif value.replace(".", "", 1).isdigit():
                passed_value[key] = float(value)



SUMMARY:
**Summary:**

The `_convert_str_dict` function takes a dictionary as input, ensuring it conforms to a specific structure. It:
- Recursively converts nested dictionaries
- Converts string values to their corresponding types (bool, int, float) based on the value's format.

**Key Implementation Details:**

* Recursive conversion of nested dictionaries using `self.` method signature.
* Conversion of boolean and numeric values from strings based on specified formats.

**Notable Dependencies or Requirements:**
 
* None.


key: chroma:document
string_value:
CODE:

class ParallelMode(Enum):
    NOT_PARALLEL = "not_parallel"
    NOT_DISTRIBUTED = "not_distributed"
    DISTRIBUTED = "distributed"
    SAGEMAKER_MODEL_PARALLEL = "sagemaker_model_parallel"
    SAGEMAKER_DATA_PARALLEL = "sagemaker_data_parallel"


SUMMARY:
**Class Summary: `ParallelMode` Enum**

The `ParallelMode` class is an enumeration that defines different parallelization modes for machine learning. It provides predefined values to represent distinct scenarios, such as:

* Non-distributed processing
* Distributed data and model parallelism (Sagemaker-specific)
* TPU utilization

**Key Implementation Details:**

* Four classes define the main parallelization modes:
	+ `NOT_PARALLEL` (string "not_parallel")
	+ `NOT_DISTRIBUTED` (string "not_distributed")
	+ `DISTRIBUTED` (string "distributed")
	+ `SAGEMAKER_MODEL_PARALLEL` and `SAGEMAKER_DATA_PARALLEL` (Sagemaker-specific modes)
* Three additional constants represent TPU utilization:
	+ `TPU` (string "tpu")

**Notable Dependencies/Requirements:**

* None explicitly listed, but may interact with Sagemaker or TPU-related dependencies.


key: chroma:document
string_value:
CODE:

def seed_worker(_):
    """
    Helper function to set worker seed during Dataloader initialization.
    """
    worker_seed = torch.initial_seed() % 2**32


SUMMARY:
**Summary:**

### Function Purpose and Functionality:

`seed_worker` sets the initial seed for PyTorch's random number generator (`torch`) to ensure reproducibility in Dataloader initialization.

### Key Implementation Details:

* Calculates an integer value between 0 and 2^32 using `torch.initial_seed()`.
* Assigns this value to a worker seed using `set_seed(worker_seed)`.

### Notable Dependencies and Requirements:

* PyTorch (`torch`)
* `set_seed` function (part of PyTorch)
* Reproducibility in Dataloader initialization


key: chroma:document
string_value:
CODE:

def enable_full_determinism(seed: int, warn_only: bool = False):
    """
    Helper function for reproducible behavior during distributed training. See
    - https://pytorch.org/docs/stable/notes/randomness.html for pytorch
    - https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism for tensorflow
    """
    # set seed first
    set_seed(seed)

    if is_torch_available():
        # Enable PyTorch deterministic mode. This potentially requires either the environment
        # variable 'CUDA_LAUNCH_BLOCKING' or 'CUBLAS_WORKSPACE_CONFIG' to be set,
        # depending on the CUDA version, so we set them both here
        os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
        os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":16:8"
        # The environment variable required to enable deterministic mode on Ascend NPUs.
        os.environ["ASCEND_LAUNCH_BLOCKING"] = "1"
        os.environ["HCCL_DETERMINISTIC"] = "1"

        os.environ["FLASH_ATTENTION_DETERMINISTIC"] = "1"
        torch.use_deterministic_algorithms(True, warn_only=warn_only)

        # Enable CUDNN deterministic mode
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False

    if is_tf_available():
        import tensorflow as tf



SUMMARY:
Here's a concise summary of the function code chunk:

**`enable_full_determinism` Function**

**Main Purpose and Functionality:**
This function enables full deterministic behavior in PyTorch and/or TensorFlow for reproducible distributed training. It sets environment variables and configuration options to ensure determinism.

**Key Implementation Details:**

* Sets seed using `set_seed(seed)`.
* Enables PyTorch deterministic mode with:
	+ Setting environment variables (`CUDA_LAUNCH_BLOCKING`, `CUBLAS_WORKSPACE_CONFIG`, etc.)
	+ Using `torch.use_deterministic_algorithms(True, warn_only=warn_only)`
* Enabling CUDNN deterministic mode with `torch.backends.cudnn.deterministic = True`.
* In TensorFlow, enables op determinism using `tf.config.experimental.enable_op_determinism()`.

**Notable Dependencies and Requirements:**

* PyTorch (`torch`)
* TensorFlow (with `is_tf_available()` conditional)
* Environment variables such as:
	+ `CUDA_LAUNCH_BLOCKING`
	+ `CUBLAS_WORKSPACE_CONFIG`
	+ `ASCEND_LAUNCH_BLOCKING`
	+ `HCCL_DETERMINISTIC`
	+ `FLASH_ATTENTION_DETERMINISTIC`


key: chroma:document
string_value:
CODE:

def set_seed(seed: int, deterministic: bool = False):
    """
    Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch` and/or `tf` (if installed).

    Args:
        seed (`int`):
            The seed to set.
        deterministic (`bool`, *optional*, defaults to `False`):
            Whether to use deterministic algorithms where available. Can slow down training.
    """
    random.seed(seed)
    np.random.seed(seed)
    if is_torch_available():
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        # ^^ safe to call this function even if cuda is not available
        if deterministic:
            torch.use_deterministic_algorithms(True)
    if is_torch_mlu_available():
        torch.mlu.manual_seed_all(seed)
    if is_torch_musa_available():
        torch.musa.manual_seed_all(seed)
    if is_torch_npu_available():
        torch.npu.manual_seed_all(seed)
    if is_torch_xpu_available():
        torch.xpu.manual_seed_all(seed)
    if is_tf_available():
        import tensorflow as tf

        tf.random.set_seed(seed)
        if deterministic:


SUMMARY:
**Code Summary**

The `set_seed` function (version 0.1.0) sets seeds for various libraries:

*   Imports the necessary functions from `_random_`, `numpy_, `_torch_, and `_tensorflow_`.
*   Sets seeds using:
    *   `random.seed` (`int`)
    *   `np.random.seed` (`int`)
    *   If Torch is installed:

        -   `torch.manual_seed(seed)` (`float`, default=0.0)
            -  Safely called even without CUDA
        -   `torch.cuda.manual_seed_all(seed)` (default None, ignored if `cuda` is not available)
            -  Uses deterministic algorithms when `determinism=True`
        -   `torch.use_deterministic_algorithms(True)` (if available)
*   And checks availability and usage for various Torch versions:
    *   `_Torch MLU_`
    *   `_Torch Musa_
    *   `_Torch NPU_
    *   `_Torch XPU_
*   Sets seeds using TensorFlow's (`int`) and configures its deterministic option.

**Implementation Details**

To ensure reproducibility, this function sets the `seed` for multiple libraries (random, NumPy, Torch, etc.) in a way that prioritizes performance. The use of different techniques and functions may make the code slightly more complex but ultimately provides better coverage to guarantee certain reproducible behavior


