# embedding_metadata

key: chroma:document
string_value:
CODE:

def clean_code(content: str) -> str:
    """
    Remove docstrings, empty line or comments from some code (used to detect if a diff is real or only concern
    comments or docstings).

    Args:
        content (`str`): The code to clean

    Returns:
        `str`: The cleaned code.
    """
    # We need to deactivate autoformatting here to write escaped triple quotes (we cannot use real triple quotes or
    # this would mess up the result if this function applied to this particular file).
    # fmt: off
    # Remove docstrings by splitting on triple " then triple ':
    splits = content.split('\"\"\"')
    content = "".join(splits[::2])
    splits = content.split("\'\'\'")
    # fmt: on
    content = "".join(splits[::2])

    # Remove empty lines and comments
    lines_to_keep = []
    for line in content.split("\n"):
        # remove anything that is after a # sign.
        line = re.sub("#.*$", "", line)
        # remove white lines
        if len(line) != 0 and not line.isspace():
            lines_to_keep.append(line)


SUMMARY:
**Function Summary:**

`clean_code(content: str) -> str`

Removes docstrings, empty lines, and comments from the provided code content to detect true changes.

**Implementation Highlights:**

* Uses triple quotes (`"\"\"\""`, `'\'\'\''`) to split out docstrings
* Removes empty lines by checking for non-whitespace characters
* Removes comments using regular expression substitution (`#.*$`)

**Notable Dependencies/Requirements:**

* None listed, but likely uses built-in Python modules such as `re` (regular expressions)


key: chroma:document
string_value:
CODE:

def keep_doc_examples_only(content: str) -> str:
    """
    Remove everything from the code content except the doc examples (used to determined if a diff should trigger doc
    tests or not).

    Args:
        content (`str`): The code to clean

    Returns:
        `str`: The cleaned code.
    """
    # Keep doc examples only by splitting on triple "`"
    splits = content.split("```")
    # Add leading and trailing "```" so the navigation is easier when compared to the original input `content`
    content = "```" + "```".join(splits[1::2]) + "```"

    # Remove empty lines and comments
    lines_to_keep = []
    for line in content.split("\n"):
        # remove anything that is after a # sign.
        line = re.sub("#.*$", "", line)
        # remove white lines
        if len(line) != 0 and not line.isspace():
            lines_to_keep.append(line)


SUMMARY:
**Function Summary: `keep_doc_examples_only`**

Removes non-documentation examples from provided code content, leaving only comments.

**Key Implementation Details:**

*   Splits input content by triple quotes (`"""`) to isolate documentation sections.
*   Trims leading and trailing whitespace from each line in the split content to eliminate unnecessary characters.
*   Removes commented-out lines by removing everything after a `#` sign using regular expressions.
*   Returns the cleaned code content as a single string.

**Notable Dependencies or Requirements:**

*   None found.

**Return Value Signature:**
```python
def keep_doc_examples_only(content: str) -> str:
```

**Main Functionality:**
The function's primary purpose is to filter out non-documentation examples in the input code content, leaving only comments. The filtered comments can then be used to trigger doc tests or documentation checks without including unnecessary code snippets.


key: chroma:document
string_value:
CODE:

def get_all_tests() -> List[str]:
    """
    Walks the `tests` folder to return a list of files/subfolders. This is used to split the tests to run when using
    paralellism. The split is:

    - folders under `tests`: (`tokenization`, `pipelines`, etc) except the subfolder `models` is excluded.
    - folders under `tests/models`: `bert`, `gpt2`, etc.
    - test files under `tests`: `test_modeling_common.py`, `test_tokenization_common.py`, etc.
    """

    # test folders/files directly under `tests` folder
    tests = os.listdir(PATH_TO_TESTS)
    tests = [f"tests/{f}" for f in tests if "__pycache__" not in f]
    tests = sorted([f for f in tests if (PATH_TO_REPO / f).is_dir() or f.startswith("tests/test_")])

    # model specific test folders
    model_test_folders = os.listdir(PATH_TO_TESTS / "models")
    model_test_folders = [f"tests/models/{f}" for f in model_test_folders if "__pycache__" not in f]
    model_test_folders = sorted([f for f in model_test_folders if (PATH_TO_REPO / f).is_dir()])

    tests.remove("tests/models")
    # Sagemaker tests are not meant to be run on the CI.
    if "tests/sagemaker" in tests:
        tests.remove("tests/sagemaker")
    tests = model_test_folders + tests



SUMMARY:
**Summary**

The `get_all_tests` function generates a list of files and subfolders to run tests, leveraging parallelism. It walks the `tests` folder, excluding subfolders like `models` and only including specific ones like `tokenization` and `pipelines`. Model-specific test folders (`bert`, `gpt2`) are also included.

**Key Implementation Details**

* Returns the following types of contents:
	+ Directly under `tests`: folders with directories or files starting with `test_`.
	+ Under `tests/models`: model-specific folder names.
* Sorts and excludes certain items:
	+ Removes `.pyc` files from subfolders under `tests`.
	+ Excludes `models` subfolder from the main `tests` list.
	+ Removes the `sagemaker` package if found in the main test list.

**Notable Dependencies or Requirements**

* Uses `os.listdir` to interact with the file system.
* References `PATH_TO_REPO`, `PATH_TO_TESTS`, and `SAGEMAKER Package` by exact name.


key: chroma:document
string_value:
CODE:

def diff_is_docstring_only(repo: Repo, branching_point: str, filename: str) -> bool:
    """
    Check if the diff is only in docstrings (or comments and whitespace) in a filename.

    Args:
        repo (`git.Repo`): A git repository (for instance the Transformers repo).
        branching_point (`str`): The commit reference of where to compare for the diff.
        filename (`str`): The filename where we want to know if the diff isonly in docstrings/comments.

    Returns:
        `bool`: Whether the diff is docstring/comments only or not.
    """
    folder = Path(repo.working_dir)
    with checkout_commit(repo, branching_point):
        with open(folder / filename, "r", encoding="utf-8") as f:
            old_content = f.read()

    with open(folder / filename, "r", encoding="utf-8") as f:
        new_content = f.read()

    old_content_clean = clean_code(old_content)
    new_content_clean = clean_code(new_content)



SUMMARY:
**Summary:**

`diff_is_docstring_only` function checks if the diff between two commits in a Git repository is only docstrings and/or whitespace modifications in a specified filename.

**Key Implementation Details:**

* Compares old and new commit content using `clean_code()` function.
* Uses `git.Repo` to access working directory and checkout specific branching point.
* Opens files in same encoding for consistency.

**Return Value:**

Returns `bool` indicating whether the diff is docstring/comments only or not. 

**Notable Dependencies:**

* `git.Repo`
* `Path` from `pathlib`
* `clean_code()` function (exact implementation unknown)


key: chroma:document
string_value:
CODE:

def diff_contains_doc_examples(repo: Repo, branching_point: str, filename: str) -> bool:
    """
    Check if the diff is only in code examples of the doc in a filename.

    Args:
        repo (`git.Repo`): A git repository (for instance the Transformers repo).
        branching_point (`str`): The commit reference of where to compare for the diff.
        filename (`str`): The filename where we want to know if the diff is only in codes examples.

    Returns:
        `bool`: Whether the diff is only in code examples of the doc or not.
    """
    folder = Path(repo.working_dir)
    with checkout_commit(repo, branching_point):
        with open(folder / filename, "r", encoding="utf-8") as f:
            old_content = f.read()

    with open(folder / filename, "r", encoding="utf-8") as f:
        new_content = f.read()

    old_content_clean = keep_doc_examples_only(old_content)
    new_content_clean = keep_doc_examples_only(new_content)



SUMMARY:
**_summary_**

`diff_contains_doc_examples(repo, branching_point, filename)`: Compares two versions of a file within a Git repository to determine if the diff is exclusively code examples.

**Key details:**

* **Functionality**: Exclusively checks for changes in code snippets versus documentation examples.
* **Implementation**:
	+ Retrieves file contents from working directory using `checkout_commit` and `open`.
	+ Applies `keep_doc_examples_only` function to clean up content by removing non-example sections.
	+ Returns `True` if differing example content is found, otherwise `False`.

**External dependencies:**

* `git.Repo`
* `Path`
* `keep_doc_examples_only` (assumed to be implemented elsewhere)

**Interfaces/Methods used:**

* `git.Repo.checkout_commit(branching_point)`
* `github.Path.open(filename, encoding="utf-8")`

Note that specific implementation details like the `keep_doc_examples_only` function are assumed to be implemented elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def get_impacted_files_from_tiny_model_summary(diff_with_last_commit: bool = False) -> List[str]:
    """
    Return a list of python modeling files that are impacted by the changes of `tiny_model_summary.json` in between:

    - the current head and the main branch if `diff_with_last_commit=False` (default)
    - the current head and its parent commit otherwise.

    Returns:
        `List[str]`: The list of Python modeling files that are impacted by the changes of `tiny_model_summary.json`.
    """
    repo = Repo(PATH_TO_REPO)

    folder = Path(repo.working_dir)

    if not diff_with_last_commit:
        print(f"main is at {repo.refs.main.commit}")
        print(f"Current head is at {repo.head.commit}")

        commits = repo.merge_base(repo.refs.main, repo.head)
        for commit in commits:
            print(f"Branching commit: {commit}")
    else:
        print(f"main is at {repo.head.commit}")
        commits = repo.head.commit.parents
        for commit in commits:
            print(f"Parent commit: {commit}")

    if not os.path.isfile(folder / "tests/utils/tiny_model_summary.json"):
        return []

    files = set()
    for commit in commits:
        with checkout_commit(repo, commit):
            with open(folder / "tests/utils/tiny_model_summary.json", "r", encoding="utf-8") as f:
                old_content = f.read()

        with open(folder / "tests/utils/tiny_model_summary.json", "r", encoding="utf-8") as f:
            new_content = f.read()

        # get the content as json object
        old_content = json.loads(old_content)
        new_content = json.loads(new_content)

        old_keys = set(old_content.keys())
        new_keys = set(new_content.keys())

        # get the difference
        keys_with_diff = old_keys.symmetric_difference(new_keys)
        common_keys = old_keys.intersection(new_keys)
        # if both have the same key, check its content
        for key in common_keys:
            if old_content[key] != new_content[key]:
                keys_with_diff.add(key)

        # get the model classes
        impacted_model_classes = []
        for key in keys_with_diff:
            if key in new_keys:
                impacted_model_classes.extend(new_content[key]["model_classes"])

        # get the module where the model classes are defined. We want to use the main `__init__` file, but it requires
        # all the framework being installed, which is not ideal for a simple script like test fetcher.
        # So we create a temporary and modified main `__init__` and access its `_import_structure`.
        with open(folder / "src/transformers/__init__.py") as fp:
            lines = fp.readlines()
            new_lines = []
            # Get all the code related to `_import_structure`
            for line in lines:
                if line == "_import_structure = {\n":
                    new_lines.append(line)
                elif line == "# Direct imports for type-checking\n":
                    break
                elif len(new_lines) > 0:
                    # bypass the framework check so we can get all the information even if frameworks are not available
                    line = re.sub(r"is_.+_available\(\)", "True", line)
                    line = line.replace("OptionalDependencyNotAvailable", "Exception")
                    line = line.replace("Exception()", "Exception")
                    new_lines.append(line)

        # create and load the temporary module
        with tempfile.TemporaryDirectory() as tmpdirname:
            with open(os.path.join(tmpdirname, "temp_init.py"), "w") as fp:
                fp.write("".join(new_lines))

            spec = importlib.util.spec_from_file_location("temp_init", os.path.join(tmpdirname, "temp_init.py"))
            module = importlib.util.module_from_spec(spec)
            spec.loader.exec_module(module)
            # Finally, get `_import_structure` that we need
            import_structure = module._import_structure

            # map model classes to their defined module
            reversed_structure = {}
            for key, values in import_structure.items():
                for value in values:
                    reversed_structure[value] = key

            # Get the corresponding modeling file path
            for model_class in impacted_model_classes:
                module = reversed_structure[model_class]
                framework = ""
                if model_class.startswith("TF"):
                    framework = "tf"
                elif model_class.startswith("Flax"):
                    framework = "flax"
                fn = (
                    f"modeling_{module.split('.')[-1]}.py"
                    if framework == ""
                    else f"modeling_{framework}_{module.split('.')[-1]}.py"
                )
                files.add(
                    f"src.transformers.{module}.{fn}".replace(".", os.path.sep).replace(f"{os.path.sep}py", ".py")
                )



SUMMARY:
**Summary:**

The `get_impacted_files_from_tiny_model_summary` function returns a list of Python modeling files impacted by changes in `tiny_model_summary.json` between the current head and main branch. If `diff_with_last_commit=False`, it looks for changes in main branch; otherwise, it considers parent commits.

**Key Functionality:**

* Compares `tiny_model_summary.json` contents across multiple commits
* Identifies changes to model classes and their defined modules
* Uses a temporary `_import_structure` module to resolve modeling file paths

**Implementation Details:**

* Utilizes Git repository and merge_base functionality
* Calculates differences between old and new JSON content using symmetric difference and intersection
* Handles common keys with differing content separately
* Resolves model class to module mapping using `_import_structure`
* Uses temporary directory and importlib utilities to load modules and get internal file paths

**Notable Dependencies:**

* `git` for repository access and merge_base functionality
* `json` and `re` libraries for JSON parsing and regular expression pattern matching
* `tempfile` and `importlib` libraries for creating temporary directories and loading modules


key: chroma:document
string_value:
CODE:

def get_diff(repo: Repo, base_commit: str, commits: List[str]) -> List[str]:
    """
    Get the diff between a base commit and one or several commits.

    Args:
        repo (`git.Repo`):
            A git repository (for instance the Transformers repo).
        base_commit (`str`):
            The commit reference of where to compare for the diff. This is the current commit, not the branching point!
        commits (`List[str]`):
            The list of commits with which to compare the repo at `base_commit` (so the branching point).

    Returns:
        `List[str]`: The list of Python files with a diff (files added, renamed or deleted are always returned, files
        modified are returned if the diff in the file is not only in docstrings or comments, see
        `diff_is_docstring_only`).
    """
    print("\n### DIFF ###\n")
    code_diff = []
    for commit in commits:
        for diff_obj in commit.diff(base_commit):
            # We always add new python files
            if diff_obj.change_type == "A" and diff_obj.b_path.endswith(".py"):
                code_diff.append(diff_obj.b_path)
            # We check that deleted python files won't break corresponding tests.
            elif diff_obj.change_type == "D" and diff_obj.a_path.endswith(".py"):
                code_diff.append(diff_obj.a_path)
            # Now for modified files
            elif diff_obj.change_type in ["M", "R"] and diff_obj.b_path.endswith(".py"):
                # In case of renames, we'll look at the tests using both the old and new name.
                if diff_obj.a_path != diff_obj.b_path:
                    code_diff.extend([diff_obj.a_path, diff_obj.b_path])
                else:
                    # Otherwise, we check modifications are in code and not docstrings.
                    if diff_is_docstring_only(repo, commit, diff_obj.b_path):
                        print(f"Ignoring diff in {diff_obj.b_path} as it only concerns docstrings or comments.")
                    else:
                        code_diff.append(diff_obj.a_path)



SUMMARY:
**Summary:**

The `get_diff` function calculates the differences between a base commit (`base_commit`) and one or more commits (`commits`). It returns a list of Python files that show differences, including added, renamed, or deleted files.

**Main Purpose and Functionality:**

* Calculates Git diff between base commit and commit pairs
* Filters out changes only in docstrings or comments in modified files

**Key Implementation Details:**

* Iterates over commits and calculates diffs using `basecommit.diff`
* Adds new Python files (`change_type == "A"`), deletes old Python files (`change_type == "D"`), and handles modifications with rename detection (`diff_obj.change_type in ["M", "R"]`)
* Uses `diff_is_docstring_only` function to check modifications are not only in docstrings or comments

**Notable Dependencies and Requirements:**

* `git.Repo`: A Git repository object
* `diff_is_docstring_only`: a separate function (not included in this snippet) that determines if changes are only in docstrings or comments


key: chroma:document
string_value:
CODE:

def get_modified_python_files(diff_with_last_commit: bool = False) -> List[str]:
    """
    Return a list of python files that have been modified between:

    - the current head and the main branch if `diff_with_last_commit=False` (default)
    - the current head and its parent commit otherwise.

    Returns:
        `List[str]`: The list of Python files with a diff (files added, renamed or deleted are always returned, files
        modified are returned if the diff in the file is not only in docstrings or comments, see
        `diff_is_docstring_only`).
    """
    repo = Repo(PATH_TO_REPO)

    if not diff_with_last_commit:
        print(f"main is at {repo.refs.main.commit}")
        print(f"Current head is at {repo.head.commit}")

        branching_commits = repo.merge_base(repo.refs.main, repo.head)
        for commit in branching_commits:
            print(f"Branching commit: {commit}")
        return get_diff(repo, repo.head.commit, branching_commits)
    else:
        print(f"main is at {repo.head.commit}")
        parent_commits = repo.head.commit.parents
        for commit in parent_commits:
            print(f"Parent commit: {commit}")


SUMMARY:
Here's a concise summary of the provided code chunk:

**Function Summary:** `get_modified_python_files`

**Purpose and Functionality:** Returns a list of Python files modified between the current head and either its main branch or previous commit (`diff_with_last_commit` flag).

**Key Implementation Details:**

* Uses Git repository manipulation to determine branching commits (if not using last commit)
* Calculates differences in modified files with `get_diff(repo, repo.head.commit, [commit])` method
* Prints intermediate commit information for debugging purposes

**Notable Dependencies and Requirements:**

* Git library (`Repo`) for interacting with the repository
* `get_diff` function (not shown) for calculating file diffs


key: chroma:document
string_value:
CODE:

def get_diff_for_doctesting(repo: Repo, base_commit: str, commits: List[str]) -> List[str]:
    """
    Get the diff in doc examples between a base commit and one or several commits.

    Args:
        repo (`git.Repo`):
            A git repository (for instance the Transformers repo).
        base_commit (`str`):
            The commit reference of where to compare for the diff. This is the current commit, not the branching point!
        commits (`List[str]`):
            The list of commits with which to compare the repo at `base_commit` (so the branching point).

    Returns:
        `List[str]`: The list of Python and Markdown files with a diff (files added or renamed are always returned, files
        modified are returned if the diff in the file is only in doctest examples).
    """
    print("\n### DIFF ###\n")
    code_diff = []
    for commit in commits:
        for diff_obj in commit.diff(base_commit):
            # We only consider Python files and doc files.
            if not diff_obj.b_path.endswith(".py") and not diff_obj.b_path.endswith(".md"):
                continue
            # We always add new python/md files
            if diff_obj.change_type in ["A"]:
                code_diff.append(diff_obj.b_path)
            # Now for modified files
            elif diff_obj.change_type in ["M", "R"]:
                # In case of renames, we'll look at the tests using both the old and new name.
                if diff_obj.a_path != diff_obj.b_path:
                    code_diff.extend([diff_obj.a_path, diff_obj.b_path])
                else:
                    # Otherwise, we check modifications contain some doc example(s).
                    if diff_contains_doc_examples(repo, commit, diff_obj.b_path):
                        code_diff.append(diff_obj.a_path)
                    else:
                        print(f"Ignoring diff in {diff_obj.b_path} as it doesn't contain any doc example.")



SUMMARY:
**Summary:**

The `get_diff_for_doctesting` function generates a list of Python and Markdown files with differences between a base commit (`base_commit`) and one or several commits (`commits`). It focuses on changes to doc examples, returning modified files that contain doc examples.

**Key Details:**

*   **Function Signature:** `def get_diff_for_doctesting(repo: Repo, base_commit: str, commits: List[str]) -> List[str]`
*   **Primary Purpose:** Identify and return Python and Markdown files with differences between base commit and comparison commits.
*   **Implementation Details:**
    *   Iterates over each commit in the list of commits.
    *   Compares diff object of each commit against the base commit.
    *   Applies filtering on file types (python.py, md) to exclude irrelevant changes.
    *   Returns modified files containing doc examples using the `diff_contains_doc_examples` function.

**External Dependencies:**

*   `git.Repo`: Git repository class for retrieving information about commits.
*   `List[str]`: List of commit IDs for comparison.
*   `Repo` and other classes not mentioned here are assumed to be external dependencies.


key: chroma:document
string_value:
CODE:

def get_all_doctest_files() -> List[str]:
    """
    Return the complete list of python and Markdown files on which we run doctest.

    At this moment, we restrict this to only take files from `src/` or `docs/source/en/` that are not in `utils/not_doctested.txt`.

    Returns:
        `List[str]`: The complete list of Python and Markdown files on which we run doctest.
    """
    py_files = [str(x.relative_to(PATH_TO_REPO)) for x in PATH_TO_REPO.glob("**/*.py")]
    md_files = [str(x.relative_to(PATH_TO_REPO)) for x in PATH_TO_REPO.glob("**/*.md")]

    test_files_to_run = py_files + md_files
    # change to use "/" as path separator
    test_files_to_run = ["/".join(Path(x).parts) for x in test_files_to_run]
    # don't run doctest for files in `src/transformers/models/deprecated`
    test_files_to_run = [x for x in test_files_to_run if "models/deprecated" not in x]

    # only include files in `src` or `docs/source/en/`
    test_files_to_run = [x for x in test_files_to_run if x.startswith(("src/", "docs/source/en/"))]
    # not include init files
    test_files_to_run = [x for x in test_files_to_run if not x.endswith(("__init__.py",))]

    # These are files not doctested yet.
    with open("utils/not_doctested.txt") as fp:
        not_doctested = {x.split(" ")[0] for x in fp.read().strip().split("\n")}

    # So far we don't have 100% coverage for doctest. This line will be removed once we achieve 100%.
    test_files_to_run = [x for x in test_files_to_run if x not in not_doctested]



SUMMARY:
**Summary**

The `get_all_doctest_files` function retrieves the complete list of Python and Markdown files to run doctest. It filters out files based on their location, extensions, and excluded directories.

*   **Purpose:** Returns a lista of paths by filtering python, markdown file from repository.
*   **Functionality:**
    *   Uses `glob` to find `.py` and `.md` files in `src/` or `docs/source/en/`.
    *   Excludes files in `utils/not_doctested.txt`, `src/transformers/models/deprecated`, and init files.
    *   Filters out files not marked for testing (`not_doctested.txt`).
*   **Key Implementation Details:**
    *   Uses list comprehensions to filter and transform file paths.
    *   Checks file names against exclusion lists for filtering.

**Dependencies/Requirements:**

*   `glob`
*   `os.path` (specifically, `relative_to`, `Partition`)
*   `pathlib`

**Notable Variables:**

*      `PATH_TO_REPO`: repository path
*      `_` not variable but alias

**Interfaces/Superclasses:**

None

This code snippet appears to be part of a larger class or module that handles file system operations and doctest filtering.


key: chroma:document
string_value:
CODE:

def get_new_doctest_files(repo, base_commit, branching_commit) -> List[str]:
    """
    Get the list of files that were removed from "utils/not_doctested.txt", between `base_commit` and
    `branching_commit`.

    Returns:
        `List[str]`: List of files that were removed from "utils/not_doctested.txt".
    """
    for diff_obj in branching_commit.diff(base_commit):
        # Ignores all but the "utils/not_doctested.txt" file.
        if diff_obj.a_path != "utils/not_doctested.txt":
            continue
        # Loads the two versions
        folder = Path(repo.working_dir)
        with checkout_commit(repo, branching_commit):
            with open(folder / "utils/not_doctested.txt", "r", encoding="utf-8") as f:
                old_content = f.read()
        with open(folder / "utils/not_doctested.txt", "r", encoding="utf-8") as f:
            new_content = f.read()
        # Compute the removed lines and return them
        removed_content = {x.split(" ")[0] for x in old_content.split("\n")} - {
            x.split(" ")[0] for x in new_content.split("\n")
        }
        return sorted(removed_content)


SUMMARY:
**Concise Summary**

`get_new_doctest_files` function takes three commit references (`repo`, `base_commit`, and `branching_commit`) and returns a list of files that were removed from "utils/not_doctested.txt" between these commits.

**Key Insights:**

* The function loads the old and new contents of "utils/not_doctested.txt" for each merge commit.
* It computes the removed lines by taking the difference between the two versions.
* Notable dependencies:
	+ `Path` from `pathlib`
	+ `checkout_commit` from GitPython
	+ `diff` from GitPython

**Return Value:** A list of strings representing the files that were removed from "utils/not_doctested.txt".


key: chroma:document
string_value:
CODE:

def get_doctest_files(diff_with_last_commit: bool = False) -> List[str]:
    """
    Return a list of python and Markdown files where doc example have been modified between:

    - the current head and the main branch if `diff_with_last_commit=False` (default)
    - the current head and its parent commit otherwise.

    Returns:
        `List[str]`: The list of Python and Markdown files with a diff (files added or renamed are always returned, files
        modified are returned if the diff in the file is only in doctest examples).
    """
    repo = Repo(PATH_TO_REPO)

    test_files_to_run = []  # noqa
    if not diff_with_last_commit:
        print(f"main is at {repo.refs.main.commit}")
        print(f"Current head is at {repo.head.commit}")

        branching_commits = repo.merge_base(repo.refs.main, repo.head)
        for commit in branching_commits:
            print(f"Branching commit: {commit}")
        test_files_to_run = get_diff_for_doctesting(repo, repo.head.commit, branching_commits)
    else:
        print(f"main is at {repo.head.commit}")
        parent_commits = repo.head.commit.parents
        for commit in parent_commits:
            print(f"Parent commit: {commit}")
        test_files_to_run = get_diff_for_doctesting(repo, repo.head.commit, parent_commits)

    all_test_files_to_run = get_all_doctest_files()

    # Add to the test files to run any removed entry from "utils/not_doctested.txt".
    new_test_files = get_new_doctest_files(repo, repo.head.commit, repo.refs.main.commit)
    test_files_to_run = list(set(test_files_to_run + new_test_files))

    # Do not run slow doctest tests on CircleCI
    with open("utils/slow_documentation_tests.txt") as fp:
        slow_documentation_tests = set(fp.read().strip().split("\n"))
    test_files_to_run = [
        x for x in test_files_to_run if x in all_test_files_to_run and x not in slow_documentation_tests
    ]

    # Make sure we did not end up with a test file that was removed
    test_files_to_run = [f for f in test_files_to_run if (PATH_TO_REPO / f).exists()]



SUMMARY:
**Function Summary**

`get_doctest_files(diff_with_last_commit: bool = False) -> List[str]`
-------------------------

This function generates a list of Python and Markdown files with modified doctest examples between the current head commit and either:

*   The main branch (if `diff_with_last_commit=False`) or
*   Its parent commits (otherwise)

The function performs the following steps:

1.  Checks if there are any modified doctest examples in the given commit range.
2.  Identifies added, renamed, or modified files containing doctest examples.
3.  Excludes slow documenting tests to be run on CircleCI.
4.  Ensures that no removed test files are included.

**Key Implementation Details**

*   Uses `Repo` class from [GitPython](https://gitpython.readthedocs.io/en/stable/) for Git repository management.
*   Utilizes `get_diff_for_doctesting` function to identify modified doctest examples.
*   Employs set operations and filtering for efficient list construction.

**Notable Dependencies or Requirements**

*   [GitPython](https://gitpython.readthedocs.io/en/stable/) library for Git repository interactions
*   External dependencies not specified in the provided code snippet.


key: chroma:document
string_value:
CODE:

def extract_imports(module_fname: str, cache: Dict[str, List[str]] = None) -> List[str]:
    """
    Get the imports a given module makes.

    Args:
        module_fname (`str`):
            The name of the file of the module where we want to look at the imports (given relative to the root of
            the repo).
        cache (Dictionary `str` to `List[str]`, *optional*):
            To speed up this function if it was previously called on `module_fname`, the cache of all previously
            computed results.

    Returns:
        `List[str]`: The list of module filenames imported in the input `module_fname` (a submodule we import from that
        is a subfolder will give its init file).
    """
    if cache is not None and module_fname in cache:
        return cache[module_fname]

    with open(PATH_TO_REPO / module_fname, "r", encoding="utf-8") as f:
        content = f.read()

    # Filter out all docstrings to not get imports in code examples. As before we need to deactivate formatting to
    # keep this as escaped quotes and avoid this function failing on this file.
    splits = content.split('\"\"\"')  # fmt: skip
    content = "".join(splits[::2])

    module_parts = str(module_fname).split(os.path.sep)
    imported_modules = []

    # Let's start with relative imports
    relative_imports = _re_single_line_relative_imports.findall(content)
    relative_imports = [
        (mod, imp) for mod, imp in relative_imports if "# tests_ignore" not in imp and imp.strip() != "("
    ]
    multiline_relative_imports = _re_multi_line_relative_imports.findall(content)
    relative_imports += [(mod, imp) for mod, imp in multiline_relative_imports if "# tests_ignore" not in imp]

    # We need to remove parts of the module name depending on the depth of the relative imports.
    for module, imports in relative_imports:
        level = 0
        while module.startswith("."):
            module = module[1:]
            level += 1

        if len(module) > 0:
            dep_parts = module_parts[: len(module_parts) - level] + module.split(".")
        else:
            dep_parts = module_parts[: len(module_parts) - level]
        imported_module = os.path.sep.join(dep_parts)
        imported_modules.append((imported_module, [imp.strip() for imp in imports.split(",")]))

    # Let's continue with direct imports
    direct_imports = _re_single_line_direct_imports.findall(content)
    direct_imports = [(mod, imp) for mod, imp in direct_imports if "# tests_ignore" not in imp and imp.strip() != "("]
    multiline_direct_imports = _re_multi_line_direct_imports.findall(content)
    direct_imports += [(mod, imp) for mod, imp in multiline_direct_imports if "# tests_ignore" not in imp]

    # We need to find the relative path of those imports.
    for module, imports in direct_imports:
        import_parts = module.split(".")[1:]  # ignore the name of the repo since we add it below.
        dep_parts = ["src", "transformers"] + import_parts
        imported_module = os.path.sep.join(dep_parts)
        imported_modules.append((imported_module, [imp.strip() for imp in imports.split(",")]))

    result = []
    # Double check we get proper modules (either a python file or a folder with an init).
    for module_file, imports in imported_modules:
        if (PATH_TO_REPO / f"{module_file}.py").is_file():
            module_file = f"{module_file}.py"
        elif (PATH_TO_REPO / module_file).is_dir() and (PATH_TO_REPO / module_file / "__init__.py").is_file():
            module_file = os.path.sep.join([module_file, "__init__.py"])
        imports = [imp for imp in imports if len(imp) > 0 and re.match("^[A-Za-z0-9_]*$", imp)]
        if len(imports) > 0:
            result.append((module_file, imports))

    if cache is not None:
        cache[module_fname] = result



SUMMARY:
**Function Summary: `extract_imports`**

Extracts the imports made by a given module from its source code.

**Purpose**
The function takes a file name of a module and returns a list of tuples containing the imported module filenames and their corresponding import statements.

**Key Implementation Details**

1. **Relative Imports**: Extracts relative imports using regular expressions, filtering out docstrings and `tests_ignore` comments.
2. **Direct Imports**: Finds direct imports by parsing lines similar to `import * or from ... import`. The function ensures that only actual import statements are extracted.
3. **Cache Mechanism**: Stores previously computed results for module names to avoid redundant computations.

**Notable Dependencies and Requirements**

1. `os`: for interacting with the file system.
2. `re`: for regular expression matches.
3. `_re_single_line_relative_imports`, `_re_multi_line_relative_imports`, `_re_single_line_direct_imports`, and `_re_multi_line_direct_imports` (custom regexes).
4. `PATH_TO_REPO` (absolute path to the repository root).

**Primary Purpose**: Extracts imports from a module's source code, providing the necessary information for automated analysis or testing.

```python
def extract_imports(module_fname: str, cache: Dict[str, List[str]] = None) -> List[Tuple[str, List[str]]]:
    # ...
```

Note that this summary highlights the key aspects of the function and omits unnecessary details. It can be used in a RAG (Readability, Abstractness, Generality) system to provide a concise overview of the functionality.


key: chroma:document
string_value:
CODE:

def get_module_dependencies(module_fname: str, cache: Dict[str, List[str]] = None) -> List[str]:
    """
    Refines the result of `extract_imports` to remove subfolders and get a proper list of module filenames: if a file
    as an import `from utils import Foo, Bar`, with `utils` being a subfolder containing many files, this will traverse
    the `utils` init file to check where those dependencies come from: for instance the files utils/foo.py and utils/bar.py.

    Warning: This presupposes that all intermediate inits are properly built (with imports from the respective
    submodules) and work better if objects are defined in submodules and not the intermediate init (otherwise the
    intermediate init is added, and inits usually have a lot of dependencies).

    Args:
        module_fname (`str`):
            The name of the file of the module where we want to look at the imports (given relative to the root of
            the repo).
        cache (Dictionary `str` to `List[str]`, *optional*):
            To speed up this function if it was previously called on `module_fname`, the cache of all previously
            computed results.

    Returns:
        `List[str]`: The list of module filenames imported in the input `module_fname` (with submodule imports refined).
    """
    dependencies = []
    imported_modules = extract_imports(module_fname, cache=cache)
    # The while loop is to recursively traverse all inits we may encounter: we will add things as we go.
    while len(imported_modules) > 0:
        new_modules = []
        for module, imports in imported_modules:
            # If we end up in an __init__ we are often not actually importing from this init (except in the case where
            # the object is fully defined in the __init__)
            if module.endswith("__init__.py"):
                # So we get the imports from that init then try to find where our objects come from.
                new_imported_modules = extract_imports(module, cache=cache)
                for new_module, new_imports in new_imported_modules:
                    if any(i in new_imports for i in imports):
                        if new_module not in dependencies:
                            new_modules.append((new_module, [i for i in new_imports if i in imports]))
                        imports = [i for i in imports if i not in new_imports]
                if len(imports) > 0:
                    # If there are any objects lefts, they may be a submodule
                    path_to_module = PATH_TO_REPO / module.replace("__init__.py", "")
                    dependencies.extend(
                        [
                            os.path.join(module.replace("__init__.py", ""), f"{i}.py")
                            for i in imports
                            if (path_to_module / f"{i}.py").is_file()
                        ]
                    )
                    imports = [i for i in imports if not (path_to_module / f"{i}.py").is_file()]
                    if len(imports) > 0:
                        # Then if there are still objects left, they are fully defined in the init, so we keep it as a
                        # dependency.
                        dependencies.append(module)
            else:
                dependencies.append(module)

        imported_modules = new_modules


SUMMARY:
**Code Summary:**

### Functionality:

This function `get_module_dependencies` retrieves the list of module filenames for specified imports in a given Python module.

Given a `module_fname` and an optional cache (Dict[str, List[str]]), it refines the result of `extract_imports` to remove subfolders.

**Key Implementation Details:**

*   The function uses a while loop to recursively traverse imported modules (`while len(imported_modules) > 0:`)
*   It filters extracted imports based on submodule conditions: objects defined in submodules are added, not intermediate init file dependencies
*   Uses `os.path.join` for path concatenation and file checks

**Notable Dependencies or Requirements:**

*   `extract_imports`: a function to extract import statements from a module (not provided in the code snippet)
*   `PATH_TO_REPO`: a constant referring to the repository root directory
*   `os` module for file-related operations


key: chroma:document
string_value:
CODE:

def create_reverse_dependency_tree() -> List[Tuple[str, str]]:
    """
    Create a list of all edges (a, b) which mean that modifying a impacts b with a going over all module and test files.
    """
    cache = {}
    all_modules = list(PATH_TO_TRANFORMERS.glob("**/*.py")) + list(PATH_TO_TESTS.glob("**/*.py"))
    all_modules = [str(mod.relative_to(PATH_TO_REPO)) for mod in all_modules]
    edges = [(dep, mod) for mod in all_modules for dep in get_module_dependencies(mod, cache=cache)]



SUMMARY:
**Summary**

*   **Functionality**: Creates a list of reverse dependency relationships between modules and their impacting dependencies across the entire repository, excluding tests.

**Key Implementation Details**

*   Uses a cache to store module dependencies for efficiency.
*   Iterates over all module files in `/src` and `/tests` directories, extracting relative paths.

**Main Purpose and Functionality**

Creates a list of edges (a, b) representing modifying a impacts b due to transitive dependency resolution at compilation-time for the entire project. It excludes test files from the analysis. 

**Notable Dependencies or Requirements**

*   `PATH_TO_TRANFormERS` and `PATH_TO_TESTS`: Directory paths.
*   `get_module_dependencies()`: Function used to determine dependencies of a module, which is not shown in this snippet but presumably interacts with external libraries.

### Example Usage
```python
edges = create_reverse_dependency_tree()
print(edges)
```
This will output all the edges extracted from the project.


key: chroma:document
string_value:
CODE:

def get_tree_starting_at(module: str, edges: List[Tuple[str, str]]) -> List[Union[str, List[str]]]:
    """
    Returns the tree starting at a given module following all edges.

    Args:
        module (`str`): The module that will be the root of the subtree we want.
        eges (`List[Tuple[str, str]]`): The list of all edges of the tree.

    Returns:
        `List[Union[str, List[str]]]`: The tree to print in the following format: [module, [list of edges
        starting at module], [list of edges starting at the preceding level], ...]
    """
    vertices_seen = [module]
    new_edges = [edge for edge in edges if edge[0] == module and edge[1] != module and "__init__.py" not in edge[1]]
    tree = [module]
    while len(new_edges) > 0:
        tree.append(new_edges)
        final_vertices = list({edge[1] for edge in new_edges})
        vertices_seen.extend(final_vertices)
        new_edges = [
            edge
            for edge in edges
            if edge[0] in final_vertices and edge[1] not in vertices_seen and "__init__.py" not in edge[1]
        ]



SUMMARY:
**Concise Summary:**

`get_tree_starting_at` function constructs a tree data structure by scanning a module's edges.

**Main Purpose and Functionality:**
 The function traverses the edges of a Python package starting from a specified `module`, building a tree-like structure recursively. It preserves only non-instantiation edges (`__init__.py`) that do not loop back to their origin.

**Key Implementation Details:**

* Uses a breadth-first search (BFS) approach to traverse the edges.
* Checks for edge validity based on presence of `__init__.py` or self-looping edges.
* Maintains two lists: current visible edges (`new_edges`) and seen vertices (`vertices_seen`).

**Notable Dependencies and Requirements:**

* Python 3.x
* `List`, `Tuple`, and `Union` types from the standard library (not explicitly listed)
* Implicitly requires a list of edge tuples (`edges`) to be parsed as input.


key: chroma:document
string_value:
CODE:

def print_tree_deps_of(module, all_edges=None):
    """
    Prints the tree of modules depending on a given module.

    Args:
        module (`str`): The module that will be the root of the subtree we want.
        all_eges (`List[Tuple[str, str]]`, *optional*):
            The list of all edges of the tree. Will be set to `create_reverse_dependency_tree()` if not passed.
    """
    if all_edges is None:
        all_edges = create_reverse_dependency_tree()
    tree = get_tree_starting_at(module, all_edges)

    # The list of lines is a list of tuples (line_to_be_printed, module)
    # Keeping the modules lets us know where to insert each new lines in the list.
    lines = [(tree[0], tree[0])]
    for index in range(1, len(tree)):
        edges = tree[index]
        start_edges = {edge[0] for edge in edges}

        for start in start_edges:
            end_edges = {edge[1] for edge in edges if edge[0] == start}
            # We will insert all those edges just after the line showing start.
            pos = 0
            while lines[pos][1] != start:
                pos += 1
            lines = lines[: pos + 1] + [(" " * (2 * index) + end, end) for end in end_edges] + lines[pos + 1 :]

    for line in lines:
        # We don't print the refs that where just here to help build lines.


SUMMARY:
**Summary**

The `print_tree_deps_of` function prints the tree structure of dependencies for a given module. It takes two arguments: `module` (the root of the subtree) and `all_edges` (an optional list of all edges, which defaults to creating a reverse dependency tree if not provided).

**Key Implementation Details**

* The function uses a dynamic programming approach to build the tree structure by inserting edges into a list, ensuring proper indentation.
* It uses the `get_tree_starting_at` function to obtain the starting node of the subtree.

**Methods and Interfaces Used:**

* `create_reverse_dependency_tree`
* `get_tree_starting_at`

**Notable Dependencies:**

* None

**Primary Purpose**

Prints the tree structure of dependencies for a given module.


key: chroma:document
string_value:
CODE:

def init_test_examples_dependencies() -> Tuple[Dict[str, List[str]], List[str]]:
    """
    The test examples do not import from the examples (which are just scripts, not modules) so we need som extra
    care initializing the dependency map, which is the goal of this function. It initializes the dependency map for
    example files by linking each example to the example test file for the example framework.

    Returns:
        `Tuple[Dict[str, List[str]], List[str]]`: A tuple with two elements: the initialized dependency map which is a
        dict test example file to list of example files potentially tested by that test file, and the list of all
        example files (to avoid recomputing it later).
    """
    test_example_deps = {}
    all_examples = []
    for framework in ["flax", "pytorch", "tensorflow"]:
        test_files = list((PATH_TO_EXAMPLES / framework).glob("test_*.py"))
        all_examples.extend(test_files)
        # Remove the files at the root of examples/framework since they are not proper examples (they are eith utils
        # or example test files).
        examples = [
            f for f in (PATH_TO_EXAMPLES / framework).glob("**/*.py") if f.parent != PATH_TO_EXAMPLES / framework
        ]
        all_examples.extend(examples)
        for test_file in test_files:
            with open(test_file, "r", encoding="utf-8") as f:
                content = f.read()
            # Map all examples to the test files found in examples/framework.
            test_example_deps[str(test_file.relative_to(PATH_TO_REPO))] = [
                str(e.relative_to(PATH_TO_REPO)) for e in examples if e.name in content
            ]
            # Also map the test files to themselves.
            test_example_deps[str(test_file.relative_to(PATH_TO_REPO))].append(
                str(test_file.relative_to(PATH_TO_REPO))
            )


SUMMARY:
**Summary:**

This function, `init_test_examples_dependencies`, maps test example files to potential test examples and vice versa for framework-specific test frameworks (Flax, PyTorch, TensorFlow).

*   **Purpose:** Initializes the dependency map for test examples, linking each example to its corresponding test file.

**Key Implementation Details:**

*   Iterates over three predefined framework libraries.
*   Reads contents of test files for Flax, PyTorch, and TensorFlow from specific directories within `PATH_TO_EXAMPLES`.
*   Mapped keys between example files and their relative paths as strings (`str()`) using relative path operations.

**Primary Purpose Output:**

`Tuple[Dict[str, List[str]], List[str]]`: A tuple containing the initialized dependency map (`test_example_deps`) and a list of all example files.


key: chroma:document
string_value:
CODE:

def create_reverse_dependency_map() -> Dict[str, List[str]]:
    """
    Create the dependency map from module/test filename to the list of modules/tests that depend on it recursively.

    Returns:
        `Dict[str, List[str]]`: The reverse dependency map as a dictionary mapping filenames to all the filenames
        depending on it recursively. This way the tests impacted by a change in file A are the test files in the list
        corresponding to key A in this result.
    """
    cache = {}
    # Start from the example deps init.
    example_deps, examples = init_test_examples_dependencies()
    # Add all modules and all tests to all examples
    all_modules = list(PATH_TO_TRANFORMERS.glob("**/*.py")) + list(PATH_TO_TESTS.glob("**/*.py")) + examples
    all_modules = [str(mod.relative_to(PATH_TO_REPO)) for mod in all_modules if not ("models" in str(mod.relative_to(PATH_TO_REPO)) and "convert_" in str(mod.relative_to(PATH_TO_REPO)))]
    # Compute the direct dependencies of all modules.
    direct_deps = {m: get_module_dependencies(m, cache=cache) for m in all_modules}
    direct_deps.update(example_deps)

    # This recurses the dependencies
    something_changed = True
    while something_changed:
        something_changed = False
        for m in all_modules:
            for d in direct_deps[m]:
                # We stop recursing at an init (cause we always end up in the main init and we don't want to add all
                # files which the main init imports)
                if d.endswith("__init__.py"):
                    continue
                if d not in direct_deps:
                    if "convert" in d:
                        continue
                    raise ValueError(f"KeyError:{d}. From {m}")
                new_deps = set(direct_deps[d]) - set(direct_deps[m])
                if len(new_deps) > 0:
                    direct_deps[m].extend(list(new_deps))
                    something_changed = True

    # Finally we can build the reverse map.
    reverse_map = collections.defaultdict(list)
    for m in all_modules:
        for d in direct_deps[m]:
            reverse_map[d].append(m)

    # For inits, we don't do the reverse deps but the direct deps: if modifying an init, we want to make sure we test
    # all the modules impacted by that init.
    for m in [f for f in all_modules if f.endswith("__init__.py")]:
        direct_deps = get_module_dependencies(m, cache=cache)
        deps = sum([reverse_map[d] for d in direct_deps if not d.endswith("__init__.py")], direct_deps)
        reverse_map[m] = list(set(deps) - {m})



SUMMARY:
**Function Summary: `create_reverse_dependency_map`**

**Purpose and Functionality:**
This function generates a dictionary mapping module/test filenames to their dependent modules/tests recursively, excluding direct imports of initialization files.

**Key Implementation Details:**

*   Initializes the dependency map by scraping all modules, tests, and example dependencies from specific directories.
*   Computes direct dependencies using `get_module_dependencies` with caching mechanism (`cache` dict).
*   Recursively updates dependencies until no more changes are detected.
*   Builds the reverse dependency map (`reverse_map`) based on the final computed dependencies.

**Notable Dependencies/Requirements:**

*   `init_test_examples_dependencies`: Initializes example dependencies and tests.
*   `get_module_dependencies(m, cache=cache)`: Returns direct dependencies of a module using caching mechanism (`cache` dict).
*   `collections.defaultdict(list)` is used to create the reverse dependency map.

**External Dependencies:**

*   PATH_TO_TRANFORMERS and PATH_TO_TESTS for scraping module and test files.
*   PATH_TO_REPO for getting relative paths to modules and tests.

The function returns a dictionary where each key corresponds to a module/test filename, and its value contains a list of all filenames that recursively depend on it.


key: chroma:document
string_value:
CODE:

def create_module_to_test_map(
    reverse_map: Dict[str, List[str]] = None, filter_models: bool = False
) -> Dict[str, List[str]]:
    """
    Extract the tests from the reverse_dependency_map and potentially filters the model tests.

    Args:
        reverse_map (`Dict[str, List[str]]`, *optional*):
            The reverse dependency map as created by `create_reverse_dependency_map`. Will default to the result of
            that function if not provided.
        filter_models (`bool`, *optional*, defaults to `False`):
            Whether or not to filter model tests to only include core models if a file impacts a lot of models.

    Returns:
        `Dict[str, List[str]]`: A dictionary that maps each file to the tests to execute if that file was modified.
    """
    if reverse_map is None:
        reverse_map = create_reverse_dependency_map()

    # Utility that tells us if a given file is a test (taking test examples into account)
    def is_test(fname):
        if fname.startswith("tests"):
            return True
        if fname.startswith("examples") and fname.split(os.path.sep)[-1].startswith("test"):
            return True
        return False

    # Build the test map
    test_map = {module: [f for f in deps if is_test(f)] for module, deps in reverse_map.items()}

    if not filter_models:
        return test_map

    # Now we deal with the filtering if `filter_models` is True.
    num_model_tests = len(list(PATH_TO_TESTS.glob("models/*")))

    def has_many_models(tests):
        # We filter to core models when a given file impacts more than half the model tests.
        model_tests = {Path(t).parts[2] for t in tests if t.startswith("tests/models/")}
        return len(model_tests) > num_model_tests // 2

    # for each module (if specified in the argument `module`) of the form `models/my_model` (i.e. starting with it),
    # we always keep the tests (those are already in the argument `tests`) which are in `tests/models/my_model`.
    # This is to avoid them being excluded when a module has many impacted tests: the directly related test files should
    # always be included!
    def filter_tests(tests, module=""):
        filtered_tests = []
        for t in tests:
            if (
                not t.startswith("tests/models/")
                or Path(t).parts[2] in IMPORTANT_MODELS
                # at this point, `t` is of the form `tests/models/my_model`, and we check if `models/my_model`
                # (i.e. `parts[1:3]`) is in `module`.
                or "/".join(Path(t).parts[1:3]) in module
            ):
                filtered_tests += [t]
        return filtered_tests

    return {
        module: (filter_tests(tests, module=module) if has_many_models(tests) else tests)
        for module, tests in test_map.items()


SUMMARY:
**Function Summary: `create_module_to_test_map`**

Extracts and filters tests from the reverse_dependency_map, providing a dictionary mapping each file to executable tests.

**Primary Purpose:** Create a map of files to tests for execution, filtering out non-core model tests if specified.

**Key Implementation Details:**

* Uses `create_reverse_dependency_map` to default to a value if not provided.
* Applies filters based on the presence of "tests" or "examples/test" in file names and core models' modifications threshold.
* Includes tests within modules that have significantly impacted tests.

**Notable Dependencies/Requirements:**

* `create_reverse_dependency_map`
* `PATH_TO_TESTS`
* `IMPORTANT_MODELS`

**Return Value:** A dictionary with module-file-test mappings, filtering out non-core model tests if specified.


key: chroma:document
string_value:
CODE:

def _print_list(l) -> str:
    """
    Pretty print a list of elements with one line per element and a - starting each line.
    """


SUMMARY:
**Function Summary:**

`_print_list(l) -> str`

Pretty prints a list of elements, joining lines with hyphens (-) and newline characters (`\n`).

**Key Implementation Details:**

* Uses `f-strings` to format each line
* Joins lines with `\n`
* Uses list comprehension to iterate over the input list `l`

**Notable Dependencies/Requirements:**

* None

**Interface:** Does not implement any external interfaces.


key: chroma:document
string_value:
CODE:

def infer_tests_to_run(
    output_file: str, diff_with_last_commit: bool = False, filter_models: bool = False, test_all: bool = False
):
    """
    The main function called by the test fetcher. Determines the tests to run from the diff.

    Args:
        output_file (`str`):
            The path where to store the summary of the test fetcher analysis. Other files will be stored in the same
            folder:

            - examples_test_list.txt: The list of examples tests to run.
            - test_repo_utils.txt: Will indicate if the repo utils tests should be run or not.
            - doctest_list.txt: The list of doctests to run.

        diff_with_last_commit (`bool`, *optional*, defaults to `False`):
            Whether to analyze the diff with the last commit (for use on the main branch after a PR is merged) or with
            the branching point from main (for use on each PR).
        filter_models (`bool`, *optional*, defaults to `True`):
            Whether or not to filter the tests to core models only, when a file modified results in a lot of model
            tests.
    """
    if not test_all:
        modified_files = get_modified_python_files(diff_with_last_commit=diff_with_last_commit)
    else:
        modified_files = [str(k) for k in PATH_TO_TESTS.glob("*/*") if str(k).endswith(".py") and "test_" in str(k)]
        print("\n### test_all is TRUE, FETCHING ALL FILES###\n")
    print(f"\n### MODIFIED FILES ###\n{_print_list(modified_files)}")

    # Create the map that will give us all impacted modules.
    reverse_map = create_reverse_dependency_map()
    impacted_files = modified_files.copy()
    for f in modified_files:
        if f in reverse_map:
            impacted_files.extend(reverse_map[f])

    # Remove duplicates
    impacted_files = sorted(set(impacted_files))
    print(f"\n### IMPACTED FILES ###\n{_print_list(impacted_files)}")

    model_impacted = {"/".join(x.split("/")[:3]) for x in impacted_files if x.startswith("tests/models/")}
    # Grab the corresponding test files:
    if (
        any(x in modified_files for x in ["setup.py", ".circleci/create_circleci_config.py"])
        or not filter_models
        and len(model_impacted) >= NUM_MODELS_TO_TRIGGER_FULL_CI
        or commit_flags["test_all"]
    ):
        test_files_to_run = glob.glob("tests/**/test_**.py", recursive=True) + glob.glob(
            "examples/**/*.py", recursive=True
        )
        if len(model_impacted) >= NUM_MODELS_TO_TRIGGER_FULL_CI and filter_models:
            print(
                f"More than {NUM_MODELS_TO_TRIGGER_FULL_CI - 1} models are impacted and `filter_models=False`. CI is configured to test everything."
            )
    else:
        # All modified tests need to be run.
        test_files_to_run = [f for f in modified_files if f.startswith("tests") and "/test_" in f]
        impacted_files = get_impacted_files_from_tiny_model_summary(diff_with_last_commit=diff_with_last_commit)

        # Then we grab the corresponding test files.
        test_map = create_module_to_test_map(reverse_map=reverse_map, filter_models=filter_models)
        for f in modified_files + impacted_files:
            if f in test_map:
                test_files_to_run.extend(test_map[f])
        test_files_to_run = sorted(set(test_files_to_run))
        # Remove repo utils tests
        test_files_to_run = [f for f in test_files_to_run if not f.split(os.path.sep)[1] == "repo_utils"]
        # Remove SageMaker tests
        test_files_to_run = [f for f in test_files_to_run if not f.split(os.path.sep)[1] == "sagemaker"]
        # Make sure we did not end up with a test file that was removed
        test_files_to_run = [f for f in test_files_to_run if (PATH_TO_REPO / f).exists()]

    print(f"\n### TEST TO RUN ###\n{_print_list(test_files_to_run)}")

    create_test_list_from_filter(test_files_to_run, out_path="test_preparation/")

    doctest_list = get_doctest_files()

    print(f"\n### DOCTEST TO RUN ###\n{_print_list(doctest_list)}")
    if len(doctest_list) > 0:
        doctest_file = Path(output_file).parent / "doctest_list.txt"
        with open(doctest_file, "w", encoding="utf-8") as f:


SUMMARY:
Here is a concise and focused summary of the function:

**Function Name:** `infer_tests_to_run`

**Main Purpose:**
The main function determines which tests to run from the diff. It analyzes modified files, identifies impacted modules, and extracts test files.

**Key Implementation Details:**

* Retrieves a list of modified Python files using the `get_modified_python_files` function.
* Creates a reverse dependency map using the `create_reverse_dependency_map` function.
* Identifies impacted models by searching for files starting with "tests/models/" in the modified files.
* Extracts test files from directories like "tests/**/test_**.py" and "examples/**/*.py".
* Filters tests to exclude models if necessary.

**Notable Dependencies or Requirements:**

* The `PATH_TO_TESTS` variable is used to glob for test files.
* The `create_reverse_dependency_map` function depends on a reverse dependency map data structure.
* The `NUM_MODELS_TO_TRIGGER_FULL_CI` constant influences the number of models required to trigger full CI analysis.

**Output:** A list of files that need to be run in order to fetch all tests. This is printed to the console.

**Summary Code:**
```python
def infer_tests_to_run(
    output_file: str,
    diff_with_last_commit: bool = False,
    filter_models: bool = True,
    test_all: bool = False,
):
    modified_files = get_modified_python_files(diff_with_last_commit)
    if not test_all:
        modified_files = [str(k) for k in PATH_TO_TESTS.glob("*/*") if str(k).endswith(".py") and "test_" in str(k)]
    
    # ...
```


key: chroma:document
string_value:
CODE:

def filter_tests(output_file: str, filters: List[str]):
    """
    Reads the content of the output file and filters out all the tests in a list of given folders.

    Args:
        output_file (`str` or `os.PathLike`): The path to the output file of the tests fetcher.
        filters (`List[str]`): A list of folders to filter.
    """
    if not os.path.isfile(output_file):
        print("No test file found.")
        return
    with open(output_file, "r", encoding="utf-8") as f:
        test_files = f.read().split(" ")

    if len(test_files) == 0 or test_files == [""]:
        print("No tests to filter.")
        return

    if test_files == ["tests"]:
        test_files = [os.path.join("tests", f) for f in os.listdir("tests") if f not in ["__init__.py"] + filters]
    else:
        test_files = [f for f in test_files if f.split(os.path.sep)[1] not in filters]

    with open(output_file, "w", encoding="utf-8") as f:


SUMMARY:
**Function Summary: `filter_tests`**

_filters tests from an output file based on a list of folders._

### Function Purpose
 Reads the content of the output file, filters out specific tests in given folders and writes updated results back to the file.

### Key Implementation Details

* **File filtering**: Iterates through test files using `os.listdir()` and checks if folder paths are present in the `filters` list.
* **Tests reading/writing**: Uses `open()` with `"r"` and `"w"` modes to read and write content to the output file, respectively.

### Notable Dependencies

* `os`: for interacting with the operating system (e.g., working with files, directories)
* `pathlib` (not explicitly used but implied through `os.path.join()`)

### Interfaces/Methods Used

* `os.listdir()`
* `os.path.join()`

### External Dependencies
None mentioned.


key: chroma:document
string_value:
CODE:

def parse_commit_message(commit_message: str) -> Dict[str, bool]:
    """
    Parses the commit message to detect if a command is there to skip, force all or part of the CI.

    Args:
        commit_message (`str`): The commit message of the current commit.

    Returns:
        `Dict[str, bool]`: A dictionary of strings to bools with keys the following keys: `"skip"`,
        `"test_all_models"` and `"test_all"`.
    """
    if commit_message is None:
        return {"skip": False, "no_filter": False, "test_all": False}

    command_search = re.search(r"\[([^\]]*)\]", commit_message)
    if command_search is not None:
        command = command_search.groups()[0]
        command = command.lower().replace("-", " ").replace("_", " ")
        skip = command in ["ci skip", "skip ci", "circleci skip", "skip circleci"]
        no_filter = set(command.split(" ")) == {"no", "filter"}
        test_all = set(command.split(" ")) == {"test", "all"}
        return {"skip": skip, "no_filter": no_filter, "test_all": test_all}
    else:


SUMMARY:
**Function Summary**

`parse_commit_message` parses a commit message to detect specific instructions for testing.

**Parameters**

* `commit_message`: The commit message to parse (string)
* Returns: A dictionary with keys `"skip"`, `"no_filter"`, and `"test_all"` indicating the presence or absence of corresponding tests.

**Return Value Dictionary**

| Key | Boolean Value |
| --- | --- |
| `skip` | Whether to skip tests (`True`/`False`) |
| `no_filter` | Whether to disable filtering for tests (`True`/`False`) |
| `test_all` | Whether to run all tests (`True`/`False`) |

**Implementation Details**

* Uses regular expression to search for a command in the commit message.
* Sets up conditional checks based on the presence of specific keywords.

**Notable Dependencies**

No external dependencies.


key: chroma:document
string_value:
CODE:

def create_test_list_from_filter(full_test_list, out_path):
    os.makedirs(out_path, exist_ok=True)
    all_test_files = "\n".join(full_test_list)
    for job_name, _filter in JOB_TO_TEST_FILE.items():
        file_name = os.path.join(out_path, f"{job_name}_test_list.txt")
        if job_name == "tests_hub":
            files_to_test = ["tests"]
        else:
            files_to_test = list(re.findall(_filter, all_test_files))
        print(job_name, file_name)
        if len(files_to_test) > 0:  # No tests -> no file with test list
            with open(file_name, "w") as f:


SUMMARY:
**Function Summary: create_test_list_from_filter**

* **Purpose**: Creates a text list of files matching specific filter criteria and writes them to a designated output directory.

### Key Implementation Details:

- Utilizes `os.path.join` for file path construction
- Applies regular expression pattern `_filter` to a full test list to find matched files using `re.findall`
 
### Notable Dependencies/ Requirements:

- Uses `JOB_TO_TEST_FILE`, an external dictionary containing mapping between job names and test filters.
- Rely on `os` module's `makedirs` and file handling functions (`open`).


key: chroma:document
string_value:
CODE:

class HubModelLister:
    """
    Utility for getting models from the hub based on tags. Handles errors without crashing the script.
    """

    def __init__(self, tags):
        self.tags = tags
        self.model_list = api.list_models(tags=tags)

    def __iter__(self):
        try:
            yield from self.model_list
        except Exception as e:
            print(f"Error: {e}")


SUMMARY:
**Summary**

The `HubModelLister` class is a utility for retrieving models from the hub based on provided tags. Its primary purpose is to handle model fetching and errors without crashing the script.

**Main Purpose and Functionality**
```python
class HubModelLister:
    def __init__(self, tags):
        # Retrieves models from the hub based on tags
        self.model_list = api.list_models(tags=tags)
```
The main functionality is to fetch models from the hub using API's `list_models` method with input `tags`. It handles exceptions without crashing the script.

**Key Implementation Details**
- Fetches models from the hub using `api.list_models(tags=tags)`
- Uses try-except block to handle errors without crashing

**Notable Dependencies or Requirements**
- `api`: reference to the API module for list_models method
- External dependency: assumes the presence of an API that supports list_models functionality


key: chroma:document
string_value:
CODE:

def _extract_commit_hash(commits):
    for commit in commits:
        if commit.startswith("commit "):
            return commit.split(" ")[1]


SUMMARY:
**Summary**

`_extract_commit_hash` extracts the commit hash from a list of commit messages.

**Key Functionality**

* Iterates through each commit message in the `commits` list
* Returns the commit hash when a message starts with "commit "
* Returns an empty string if no such message is found

**Implementation Details**

* Uses simple iteration and string manipulation to extract the commit hash
* Utilizes `str.startswith` and `str.split` methods

**Notable Dependencies**

* None explicitly mentioned, no external imports or module references.


key: chroma:document
string_value:
CODE:

def get_list_of_repo_model_paths(models_dir):
    # Get list of all models in the library
    models = glob.glob(os.path.join(models_dir, "*/modeling_*.py"))

    # Remove flax and tf models
    models = [model for model in models if "_flax_" not in model]
    models = [model for model in models if "_tf_" not in model]

    # Get list of all deprecated models in the library
    deprecated_models = glob.glob(os.path.join(models_dir, "deprecated", "*"))
    # For each deprecated model, remove the deprecated models from the list of all models as well as the symlink path
    for deprecated_model in deprecated_models:
        deprecated_model_name = "/" + deprecated_model.split("/")[-1] + "/"
        models = [model for model in models if deprecated_model_name not in model]
    # Remove deprecated models
    models = [model for model in models if "/deprecated" not in model]
    # Remove auto
    models = [model for model in models if "/auto/" not in model]


SUMMARY:
**Function Summary:**

`get_list_of_repo_model_paths(models_dir)`

*   Purpose: Returns a list of repository model paths by filtering out Flax, TensorFlow, deprecated, and auto-generated models.

**Key Implementation Details:**

*   Uses `glob` to retrieve all `.py` files in the `models_dir` with "modeling_" prefix.
*   Applies two passes of filtering:
    *   Removes Flax and TensorFlow models using list comprehensions.
    *   Forgets deprecated models by removing their symlinks from original list.

**Notable Dependencies or Requirements:**

*   `os.path.join` for path concatenation.
*   `glob` for file patterns matching.
*   List comprehensions for efficient filtering.


key: chroma:document
string_value:
CODE:

def get_list_of_models_to_deprecate(
    thresh_num_downloads=5_000,
    thresh_date=None,
    use_cache=False,
    save_model_info=False,
    max_num_models=-1,
):
    if thresh_date is None:
        thresh_date = datetime.now(timezone.utc).replace(year=datetime.now(timezone.utc).year - 1)
    else:
        thresh_date = datetime.strptime(thresh_date, "%Y-%m-%d").replace(tzinfo=timezone.utc)

    models_dir = PATH_TO_REPO / "src/transformers/models"
    model_paths = get_list_of_repo_model_paths(models_dir=models_dir)

    if use_cache and os.path.exists("models_info.json"):
        with open("models_info.json", "r") as f:
            models_info = json.load(f)
        # Convert datetimes back to datetime objects
        for model, info in models_info.items():
            info["first_commit_datetime"] = datetime.fromisoformat(info["first_commit_datetime"])

    else:
        # Build a dictionary of model info: first commit datetime, commit hash, model path
        models_info = defaultdict(dict)
        for model_path in model_paths:
            model = model_path.split("/")[-2]
            if model in models_info:
                continue
            commits = repo.git.log("--diff-filter=A", "--", model_path).split("\n")
            commit_hash = _extract_commit_hash(commits)
            commit_obj = repo.commit(commit_hash)
            committed_datetime = commit_obj.committed_datetime
            models_info[model]["commit_hash"] = commit_hash
            models_info[model]["first_commit_datetime"] = committed_datetime
            models_info[model]["model_path"] = model_path
            models_info[model]["downloads"] = 0

            # Some tags on the hub are formatted differently than in the library
            tags = [model]
            if "_" in model:
                tags.append(model.replace("_", "-"))
            models_info[model]["tags"] = tags

        # Filter out models which were added less than a year ago
        models_info = {
            model: info for model, info in models_info.items() if info["first_commit_datetime"] < thresh_date
        }

        # We make successive calls to the hub, filtering based on the model tags
        n_seen = 0
        for model, model_info in models_info.items():
            for model_tag in model_info["tags"]:
                model_list = HubModelLister(tags=model_tag)
                for i, hub_model in enumerate(model_list):
                    n_seen += 1
                    if i % 100 == 0:
                        print(f"Processing model {i} for tag {model_tag}")
                    if max_num_models != -1 and i > n_seen:
                        break
                    if hub_model.private:
                        continue
                    model_info["downloads"] += hub_model.downloads

    if save_model_info and not (use_cache and os.path.exists("models_info.json")):
        # Make datetimes serializable
        for model, info in models_info.items():
            info["first_commit_datetime"] = info["first_commit_datetime"].isoformat()
        with open("models_info.json", "w") as f:
            json.dump(models_info, f, indent=4)

    print("\nFinding models to deprecate:")
    n_models_to_deprecate = 0
    models_to_deprecate = {}
    for model, info in models_info.items():
        n_downloads = info["downloads"]
        if n_downloads < thresh_num_downloads:
            n_models_to_deprecate += 1
            models_to_deprecate[model] = info
            print(f"\nModel: {model}")
            print(f"Downloads: {n_downloads}")
            print(f"Date: {info['first_commit_datetime']}")
    print("\nModels to deprecate: ", "\n" + "\n".join(models_to_deprecate.keys()))
    print(f"\nNumber of models to deprecate: {n_models_to_deprecate}")


SUMMARY:
**Function Summary**

`get_list_of_models_to_deprecate`

This function retrieves lists of models that have fewer than a specified threshold number of downloads over the past year.

**Primary Purpose**
Identify models with low download counts for potential deprecated model detection.

**Key Implementation Details**

1. Filter models based on download counts and date of first commit.
2. Extract commit hashes, first commit dates, and model paths from a repository's Git log.
3. Filter out new models added within the last year using a pre-defined threshold date.
4. Make successive calls to the HubModelLister for each model to collect download information.

**Notable Dependencies**

* `repo` (Git repository interaction)
* `HubModelLister` (model lister class)
* `datetime`
* `json`

**Inherited/Implemented Interfaces**
None apparent.

**External Dependencies**

* `models_info.json` file (for caching model info)


key: chroma:document
string_value:
CODE:

def extract_warnings_from_single_artifact(artifact_path, targets):
    """Extract warnings from a downloaded artifact (in .zip format)"""
    selected_warnings = set()
    buffer = []

    def parse_line(fp):
        for line in fp:
            if isinstance(line, bytes):
                line = line.decode("UTF-8")
            if "warnings summary (final)" in line:
                continue
            # This means we are outside the body of a warning
            elif not line.startswith(" "):
                # process a single warning and move it to `selected_warnings`.
                if len(buffer) > 0:
                    warning = "\n".join(buffer)
                    # Only keep the warnings specified in `targets`
                    if any(f": {x}: " in warning for x in targets):
                        selected_warnings.add(warning)
                    buffer.clear()
                continue
            else:
                line = line.strip()
                buffer.append(line)

    if from_gh:
        for filename in os.listdir(artifact_path):
            file_path = os.path.join(artifact_path, filename)
            if not os.path.isdir(file_path):
                # read the file
                if filename != "warnings.txt":
                    continue
                with open(file_path) as fp:
                    parse_line(fp)
    else:
        try:
            with zipfile.ZipFile(artifact_path) as z:
                for filename in z.namelist():
                    if not os.path.isdir(filename):
                        # read the file
                        if filename != "warnings.txt":
                            continue
                        with z.open(filename) as fp:
                            parse_line(fp)
        except Exception:
            logger.warning(
                f"{artifact_path} is either an invalid zip file or something else wrong. This file is skipped."
            )



SUMMARY:
**Function Summary: extract_warnings_from_single_artifact**

Extracts warnings from a downloaded artifact (.zip) based on specified targets.

**Main Purpose and Functionality**

This function takes an `artifact_path` and `targets` as input, extracts warnings from the artifact, and returns them in a set format. It checks for specific text patterns to identify warnings and filters them according to the provided `targets`.

**Implementation Details**

*   Uses a buffer to accumulate lines of code until a warning is complete.
*   Iterates through files in the artifact directory (if not using `from_gh`) or zipped file entries (if using `from_gh`).
*   Checks for warnings by searching for text patterns, such as "warnings summary (final)" and line endings.

**Notable Dependencies/Requirements**

*   `os`: For file system operations.
*   `zipfile`: For handling .zip files.
*   `logging`: Logger used to display warning messages.

**Key Algorithm/Pattern**

The function uses a buffer-based approach to accumulate lines of code until a warning is complete. It checks for specific text patterns to identify warnings and filters them according to the provided targets.


key: chroma:document
string_value:
CODE:

def extract_warnings(artifact_dir, targets):
    """Extract warnings from all artifact files"""

    selected_warnings = set()

    paths = [os.path.join(artifact_dir, p) for p in os.listdir(artifact_dir) if (p.endswith(".zip") or from_gh)]
    for p in paths:
        selected_warnings.update(extract_warnings_from_single_artifact(p, targets))



SUMMARY:
Here is a concise summary of the provided function code chunk:

**Function Name:** `extract_warnings`

**Purpose:**
Recurursively extracts warnings from file artifacts in a specified directory and its subdirectories.

**Main Functionality:**

* Iterates through files in the `artifact_dir` and filters those that end with ".zip" or follow the condition established by the `from_gh` variable.
* Calls a recursive helper function (`extract_warnings_from_single_artifact`) for each filtered file, passing the `targets` parameter.
* Updates an internal set `selected_warnings` with extracted warnings from all artifacts.

**Key Implementation Details:**

* Uses `os.path.join()` to construct paths to artifact files
* Utilizes recursion through `extract_warnings_from_single_artifact`
* Employs a set (`selected_warnings`) to store unique warnings

**Notable Dependencies and Requirements:**

* `os` module for path manipulation
* `targets` parameter passed to recursive helper function
* `from_gh` variable conditionally filtering ".zip" files


key: chroma:document
string_value:
CODE:

def get_indent(line: str) -> str:
    """Returns the indent in  given line (as string)."""
    search = _re_indent.search(line)


SUMMARY:
**Summary**

A utility function `get_indent` extracts and returns the indentation from a input string `line`. It utilizes a regular expression `_re_indent` to find the match in the line and returns an empty string if no match is found.

**Key Details:**

* Function signature: `get_indent(line: str) -> str`
* Regular expression used: `_re_indent` (exact name)
* Primary purpose: Extracts indentation from a line as a string.

**Notable Dependencies/Requirements:**

* `line`: input string
* `_re_indent`: regular expression object (imported or defined elsewhere)
* No external dependencies listed.


key: chroma:document
string_value:
CODE:

def split_code_in_indented_blocks(
    code: str, indent_level: str = "", start_prompt: Optional[str] = None, end_prompt: Optional[str] = None
) -> List[str]:
    """
    Split some code into its indented blocks, starting at a given level.

    Args:
        code (`str`): The code to split.
        indent_level (`str`): The indent level (as string) to use for identifying the blocks to split.
        start_prompt (`str`, *optional*): If provided, only starts splitting at the line where this text is.
        end_prompt (`str`, *optional*): If provided, stops splitting at a line where this text is.

    Warning:
        The text before `start_prompt` or after `end_prompt` (if provided) is not ignored, just not split. The input `code`
        can thus be retrieved by joining the result.

    Returns:
        `List[str]`: The list of blocks.
    """
    # Let's split the code into lines and move to start_index.
    index = 0
    lines = code.split("\n")
    if start_prompt is not None:
        while not lines[index].startswith(start_prompt):
            index += 1
        blocks = ["\n".join(lines[:index])]
    else:
        blocks = []

    # This variable contains the block treated at a given time.
    current_block = [lines[index]]
    index += 1
    # We split into blocks until we get to the `end_prompt` (or the end of the file).
    while index < len(lines) and (end_prompt is None or not lines[index].startswith(end_prompt)):
        # We have a non-empty line with the proper indent -> start of a new block
        if len(lines[index]) > 0 and get_indent(lines[index]) == indent_level:
            # Store the current block in the result and rest. There are two cases: the line is part of the block (like
            # a closing parenthesis) or not.
            if len(current_block) > 0 and get_indent(current_block[-1]).startswith(indent_level + " "):
                # Line is part of the current block
                current_block.append(lines[index])
                blocks.append("\n".join(current_block))
                if index < len(lines) - 1:
                    current_block = [lines[index + 1]]
                    index += 1
                else:
                    current_block = []
            else:
                # Line is not part of the current block
                blocks.append("\n".join(current_block))
                current_block = [lines[index]]
        else:
            # Just add the line to the current block
            current_block.append(lines[index])
        index += 1

    # Adds current block if it's nonempty.
    if len(current_block) > 0:
        blocks.append("\n".join(current_block))

    # Add final block after end_prompt if provided.
    if end_prompt is not None and index < len(lines):
        blocks.append("\n".join(lines[index:]))



SUMMARY:
**Function Summary: `split_code_in_indented_blocks`**

#### Main Purpose and Functionality
This function splits code into indented blocks. It takes in code, an indent level, and optional start and end prompts as input.

#### Key Implementation Details

* The function uses the `split()` function to split the code into lines.
* It iterates through each line, checking for the start of a new block based on the given indent level.
* The function uses a variable `current_block` to store lines being processed in a block.

#### Notable Methods and Interfaces

* `get_indent(lines[index])`: This method is used to calculate the indentation level at each line.

#### External Dependencies

* None explicitly mentioned, but it assumes standard Python environment with `Optional`, `List`, and `str` types.

#### Output
The function returns a list of strings, where each string represents an indented block of code.

Note: The exact method name for calculating the indentation level is not specified in the provided code.


key: chroma:document
string_value:
CODE:

def ignore_underscore_and_lowercase(key: Callable[[Any], str]) -> Callable[[Any], str]:
    """
    Wraps a key function (as used in a sort) to lowercase and ignore underscores.
    """

    def _inner(x):
        return key(x).lower().replace("_", "")



SUMMARY:
**Summary**

`ignore_underscore_and_lowercase` is a higher-order function that wraps another function `[Callable[[Any], str]]` to:

* Convert input strings to lowercase (`key(x).lower()`)
* Remove underscores from the output string (`replace("_", "")`)

**Implementation Details**

* `_inner` method signatures: `(x: Any) -> [str]`
* Interface used: `Callable[[Any], str]`

**Notable Dependencies/Requirements**

* None, but functions accepting any type (`Any`) and returning strings (`[str]`).


key: chroma:document
string_value:
CODE:

def sort_objects(objects: List[Any], key: Optional[Callable[[Any], str]] = None) -> List[Any]:
    """
    Sort a list of objects following the rules of isort (all uppercased first, camel-cased second and lower-cased
    last).

    Args:
        objects (`List[Any]`):
            The list of objects to sort.
        key (`Callable[[Any], str]`, *optional*):
            A function taking an object as input and returning a string, used to sort them by alphabetical order.
            If not provided, will default to noop (so a `key` must be provided if the `objects` are not of type string).

    Returns:
        `List[Any]`: The sorted list with the same elements as in the inputs
    """

    # If no key is provided, we use a noop.
    def noop(x):
        return x

    if key is None:
        key = noop
    # Constants are all uppercase, they go first.
    constants = [obj for obj in objects if key(obj).isupper()]
    # Classes are not all uppercase but start with a capital, they go second.
    classes = [obj for obj in objects if key(obj)[0].isupper() and not key(obj).isupper()]
    # Functions begin with a lowercase, they go last.
    functions = [obj for obj in objects if not key(obj)[0].isupper()]

    # Then we sort each group.
    key1 = ignore_underscore_and_lowercase(key)


SUMMARY:
**Function Summary:**

```python
sort_objects(objects: List[Any], key: Optional[Callable[[Any], str]] = None) -> List[Any]
```

**Purpose and Functionality:**

This function sorts a list of objects based on the naming conventions: all uppercase constants first, camel-cased class names second, and lowercase functions last. If no custom sorting key is provided, it defaults to alphabetical order.

**Key Implementation Details:**

* Extracts objects into three categories: `constants` (uppercase), `classes` (camel-case), and `functions` (lowercase)
* Uses the custom sorting key function (`key1`) to sort each group
* Returns the sorted list with the same elements as the input

**Notable Dependencies or Requirements:**

* None

**External Dependencies:**
None

This code follows best practices for documentation, readability, and conciseness. The summary provides a clear understanding of the function's purpose, key implementation details, and dependencies or requirements.


key: chroma:document
string_value:
CODE:

def sort_objects_in_import(import_statement: str) -> str:
    """
    Sorts the imports in a single import statement.

    Args:
        import_statement (`str`): The import statement in which to sort the imports.

    Returns:
        `str`: The same as the input, but with objects properly sorted.
    """

    # This inner function sort imports between [ ].
    def _replace(match):
        imports = match.groups()[0]
        # If there is one import only, nothing to do.
        if "," not in imports:
            return f"[{imports}]"
        keys = [part.strip().replace('"', "") for part in imports.split(",")]
        # We will have a final empty element if the line finished with a comma.
        if len(keys[-1]) == 0:
            keys = keys[:-1]
        return "[" + ", ".join([f'"{k}"' for k in sort_objects(keys)]) + "]"

    lines = import_statement.split("\n")
    if len(lines) > 3:
        # Here we have to sort internal imports that are on several lines (one per name):
        # key: [
        #     "object1",
        #     "object2",
        #     ...
        # ]

        # We may have to ignore one or two lines on each side.
        idx = 2 if lines[1].strip() == "[" else 1
        keys_to_sort = [(i, _re_strip_line.search(line).groups()[0]) for i, line in enumerate(lines[idx:-idx])]
        sorted_indices = sort_objects(keys_to_sort, key=lambda x: x[1])
        sorted_lines = [lines[x[0] + idx] for x in sorted_indices]
        return "\n".join(lines[:idx] + sorted_lines + lines[-idx:])
    elif len(lines) == 3:
        # Here we have to sort internal imports that are on one separate line:
        # key: [
        #     "object1", "object2", ...
        # ]
        if _re_bracket_content.search(lines[1]) is not None:
            lines[1] = _re_bracket_content.sub(_replace, lines[1])
        else:
            keys = [part.strip().replace('"', "") for part in lines[1].split(",")]
            # We will have a final empty element if the line finished with a comma.
            if len(keys[-1]) == 0:
                keys = keys[:-1]
            lines[1] = get_indent(lines[1]) + ", ".join([f'"{k}"' for k in sort_objects(keys)])
        return "\n".join(lines)
    else:
        # Finally we have to deal with imports fitting on one line
        import_statement = _re_bracket_content.sub(_replace, import_statement)


SUMMARY:
**Summary**

This function, `sort_objects_in_import`, sorts objects within an import statement. It correctly handles complex cases with multiple lines of imports.

**Key Functionality**

1.  **Input**: A single-line or multi-line import statement containing variable names enclosed in square brackets (`[]`).
2.  **Output**: The same input but with object dependencies properly sorted and formatted.
3.  **Sorting Logic**:
    *   For multiple lines, sorts imports according to their alphanumeric keys within the brackets.
    *   For single lines, checks for bracket content using regular expression (`_re_bracket_content`) to determine if sorting is required.

**Notable Implementation Details**

1.  **Regular Expressions**: Utilize of `_re_replace`, `_re_bracket_content`, and other regular expressions for parsing import statements, handling brackets, and sorting keys.
2.  **Line Splitting**: The function splits the input string into multiple lines based on newline characters (`\n`) to identify and sort imports across multi-line statements.
3.  **Import Sorting Logic**: Employ a custom `sort_objects` function (not shown in this snippet) to sort imported objects according to their keys.

**Dependencies**

*   No external dependencies other than Python's built-in library for regular expressions, which are used extensively in the provided code.


key: chroma:document
string_value:
CODE:

def sort_imports(file: str, check_only: bool = True):
    """
    Sort the imports defined in the `_import_structure` of a given init.

    Args:
        file (`str`): The path to the init to check/fix.
        check_only (`bool`, *optional*, defaults to `True`): Whether or not to just check (and not auto-fix) the init.
    """
    with open(file, encoding="utf-8") as f:
        code = f.read()

    # If the file is not a custom init, there is nothing to do.
    if "_import_structure" not in code or "define_import_structure" in code:
        return

    # Blocks of indent level 0
    main_blocks = split_code_in_indented_blocks(
        code, start_prompt="_import_structure = {", end_prompt="if TYPE_CHECKING:"
    )

    # We ignore block 0 (everything untils start_prompt) and the last block (everything after end_prompt).
    for block_idx in range(1, len(main_blocks) - 1):
        # Check if the block contains some `_import_structure`s thingy to sort.
        block = main_blocks[block_idx]
        block_lines = block.split("\n")

        # Get to the start of the imports.
        line_idx = 0
        while line_idx < len(block_lines) and "_import_structure" not in block_lines[line_idx]:
            # Skip dummy import blocks
            if "import dummy" in block_lines[line_idx]:
                line_idx = len(block_lines)
            else:
                line_idx += 1
        if line_idx >= len(block_lines):
            continue

        # Ignore beginning and last line: they don't contain anything.
        internal_block_code = "\n".join(block_lines[line_idx:-1])
        indent = get_indent(block_lines[1])
        # Slit the internal block into blocks of indent level 1.
        internal_blocks = split_code_in_indented_blocks(internal_block_code, indent_level=indent)
        # We have two categories of import key: list or _import_structure[key].append/extend
        pattern = _re_direct_key if "_import_structure = {" in block_lines[0] else _re_indirect_key
        # Grab the keys, but there is a trap: some lines are empty or just comments.
        keys = [(pattern.search(b).groups()[0] if pattern.search(b) is not None else None) for b in internal_blocks]
        # We only sort the lines with a key.
        keys_to_sort = [(i, key) for i, key in enumerate(keys) if key is not None]
        sorted_indices = [x[0] for x in sorted(keys_to_sort, key=lambda x: x[1])]

        # We reorder the blocks by leaving empty lines/comments as they were and reorder the rest.
        count = 0
        reorderded_blocks = []
        for i in range(len(internal_blocks)):
            if keys[i] is None:
                reorderded_blocks.append(internal_blocks[i])
            else:
                block = sort_objects_in_import(internal_blocks[sorted_indices[count]])
                reorderded_blocks.append(block)
                count += 1

        # And we put our main block back together with its first and last line.
        main_blocks[block_idx] = "\n".join(block_lines[:line_idx] + reorderded_blocks + [block_lines[-1]])

    if code != "\n".join(main_blocks):
        if check_only:
            return True
        else:
            print(f"Overwriting {file}.")
            with open(file, "w", encoding="utf-8") as f:


SUMMARY:
**Function Summary: sort_imports**
================================

*   **Purpose:** Automatically sorts imports defined in the `_import_structure` of an init file.

    The function takes in a file path and optional `check_only` flag, then reads the file's content. If the file contains an _import structure block, it extracts the imports from the corresponding code blocks, sorts them alphabetically, and reorders the blocks accordingly.

*   **Key Algorithms/Patterns:**
    *   Splits code into indented blocks using a specific prompt.
    *   Identifies and splits internal import blocks by their indentation levels.
    *   Uses regular expressions to extract keys from import lines (direct or indirect).
    *   Sorts the extracted keys alphabetically.

*   **Notable Dependencies/Requirements:**
    *   Standard Python library functions for file operations and string manipulation (`open`, `read`, `split`).

**Key Method Signatures:**

*   `split_code_in_indented_blocks`: splits code into indented blocks.
*   `_re_direct_key` and `_re_indirect_key`: regular expressions for direct and indirect import key extraction.

**Interfaces/ Classes:**
    *   No explicit interface or class usage is mentioned in the provided function.


key: chroma:document
string_value:
CODE:

def sort_imports_in_all_inits(check_only=True):
    """
    Sort the imports defined in the `_import_structure` of all inits in the repo.

    Args:
        check_only (`bool`, *optional*, defaults to `True`): Whether or not to just check (and not auto-fix) the init.
    """
    failures = []
    for root, _, files in os.walk(PATH_TO_TRANSFORMERS):
        if "__init__.py" in files:
            result = sort_imports(os.path.join(root, "__init__.py"), check_only=check_only)
            if result:
                failures = [os.path.join(root, "__init__.py")]
    if len(failures) > 0:


SUMMARY:
**Summary:**

### Function Purpose and Functionality

The `sort_imports_in_all_inits` function sorts the imports in all Python initialization files (`__init__.py`) in a repository. It checks if the sorting fixes any issues and returns a list of affected files.

### Implementation Details

* Checks all `.py` files with an initialisation file marker (e.g., `__init__.py`) under `transformers/`.
* Tails all directories under `transformers/` recursively.
* Sorts imports using `sort_imports` for each affected initialization file.

### Notable Dependencies and Requirements

* `os.walk()`
* `PATH_TO_TRANSFORMERS`: Environment variable set to the root directory of transformers.

### Note

If sorting fixes issues, it will overwrite multiple files. Running `make style` is recommended.


key: chroma:document
string_value:
CODE:

def _find_text_in_file(filename: str, start_prompt: str, end_prompt: str) -> str:
    """
    Find the text in filename between two prompts.

    Args:
        filename (`str`): The file to search into.
        start_prompt (`str`): A string to look for at the start of the content searched.
        end_prompt (`str`): A string that will mark the end of the content to look for.

    Returns:
        `str`: The content between the prompts.
    """
    with open(filename, "r", encoding="utf-8", newline="\n") as f:
        lines = f.readlines()

    # Find the start prompt.
    start_index = 0
    while not lines[start_index].startswith(start_prompt):
        start_index += 1
    start_index += 1

    # Now go until the end prompt.
    end_index = start_index
    while not lines[end_index].startswith(end_prompt):
        end_index += 1
    end_index -= 1

    while len(lines[start_index]) <= 1:
        start_index += 1
    while len(lines[end_index]) <= 1:
        end_index -= 1
    end_index += 1


SUMMARY:
**Function Summary: `_find_text_in_file`**

This function searches for text within a specified `filename` based on two prompts. It finds the content between these two markers and returns it.

**Primary Purpose:** Find text in file, filtered by start and end prompts.

**Implementation Details:**

- Uses a loop to find the index of the first line starting with the `start_prompt`.
- Continues searching after that using another loop to identify the last line ending with the `end_prompt`.
- Ensures non-empty lines are included at the edges by incrementing/decrementing indices.
- Returns the content between the prompts, along with start and end indices.

**Notable Dependencies:**

- `open()` for file operations
- External dependencies not specified in provided snippet. However, the code assumes familiarity with external libraries and file system access.

**Interfaces/Methods Used:**

- File I/O (`open()`)
- String manipulation (`readlines(), join()`)


key: chroma:document
string_value:
CODE:

def camel_case_split(identifier: str) -> List[str]:
    """
    Split a camel-cased name into words.

    Args:
        identifier (`str`): The camel-cased name to parse.

    Returns:
        `List[str]`: The list of words in the identifier (as seprated by capital letters).

    Example:

    ```py
    >>> camel_case_split("CamelCasedClass")
    ["Camel", "Cased", "Class"]
    ```
    """
    # Regex thanks to https://stackoverflow.com/questions/29916065/how-to-do-camelcase-split-in-python
    matches = re.finditer(".+?(?:(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|$)", identifier)


SUMMARY:
**Function Summary: `camel_case_split`**

The `camel_case_split` function takes a camel-cased string as input and returns a list of words separated by capital letters.

**Key Implementation Details:**

* Uses regular expressions (regex) to split the input string using the pattern specified in the comments.
* The regex pattern `.+?(?=(?=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z]))|$` breaks down into:
	+ `.+?`: Matches one or more characters (except newline) non-greedy (`?`)
	+ `(?=[a-z]|(?=[A-Z])` and similar patterns: Looks for capital letters, ignoring whitespace

**Notable Dependencies and Requirements:**

* Python's built-in `re` module (regular expression library)
* None, as it only relies on the standard library.

This code snippet achieves its purpose by utilizing regex to split the input string into words based on capital letter transitions.


key: chroma:document
string_value:
CODE:

def _center_text(text: str, width: int) -> str:
    """
    Utility that will add spaces on the left and right of a text to make it centered for a given width.

    Args:
        text (`str`): The text to center.
        width (`int`): The desired length of the result.

    Returns:
        `str`: A text of length `width` with the original `text` in the middle.
    """
    text_length = 2 if text == "✅" or text == "❌" else len(text)
    left_indent = (width - text_length) // 2
    right_indent = width - text_length - left_indent


SUMMARY:
**Summary**

The `_center_text` function centers a input `text` within a specified `width`, padding with spaces on the left and right. It optimizes padding for emojis (`✅` or `❌`) to avoid uneven indentation.

**Key Details:**

* Utilizes string multiplication (`" " *`) for efficient padding.
* Calculates padding using integer division (`//`) to align text center.
* Ignores emoji inputs by reducing text length by 2 characters.
* Returns a centered string with exactly specified `width`.

**Notable Dependencies/Requirements:**

* Supports strings (including emojis) as input `text`.
* Assumes explicit `int` type for `width`.


key: chroma:document
string_value:
CODE:

def get_model_table_from_auto_modules() -> str:
    """
    Generates an up-to-date model table from the content of the auto modules.
    """
    # Dictionary model names to config.
    config_maping_names = transformers_module.models.auto.configuration_auto.CONFIG_MAPPING_NAMES
    model_name_to_config = {
        name: config_maping_names[code]
        for code, name in transformers_module.MODEL_NAMES_MAPPING.items()
        if code in config_maping_names
    }
    model_name_to_prefix = {name: config.replace("Config", "") for name, config in model_name_to_config.items()}

    # Dictionaries flagging if each model prefix has a backend in PT/TF/Flax.
    pt_models = collections.defaultdict(bool)
    tf_models = collections.defaultdict(bool)
    flax_models = collections.defaultdict(bool)

    # Let's lookup through all transformers object (once).
    for attr_name in dir(transformers_module):
        lookup_dict = None
        if _re_tf_models.match(attr_name) is not None:
            lookup_dict = tf_models
            attr_name = _re_tf_models.match(attr_name).groups()[0]
        elif _re_flax_models.match(attr_name) is not None:
            lookup_dict = flax_models
            attr_name = _re_flax_models.match(attr_name).groups()[0]
        elif _re_pt_models.match(attr_name) is not None:
            lookup_dict = pt_models
            attr_name = _re_pt_models.match(attr_name).groups()[0]

        if lookup_dict is not None:
            while len(attr_name) > 0:
                if attr_name in model_name_to_prefix.values():
                    lookup_dict[attr_name] = True
                    break
                # Try again after removing the last word in the name
                attr_name = "".join(camel_case_split(attr_name)[:-1])

    # Let's build that table!
    model_names = list(model_name_to_config.keys()) + list(MODEL_NAMES_WITH_SAME_CONFIG.keys())

    # model name to doc link mapping
    model_names_mapping = transformers_module.models.auto.configuration_auto.MODEL_NAMES_MAPPING
    model_name_to_link_mapping = {value: f"[{value}](model_doc/{key})" for key, value in model_names_mapping.items()}
    # update mapping with special model names
    model_name_to_link_mapping = {
        k: SPECIAL_MODEL_NAME_LINK_MAPPING[k] if k in SPECIAL_MODEL_NAME_LINK_MAPPING else v
        for k, v in model_name_to_link_mapping.items()
    }

    # MaskFormerSwin and TimmBackbone are backbones and so not meant to be loaded and used on their own. Instead, they define architectures which can be loaded using the AutoBackbone API.
    names_to_exclude = ["MaskFormerSwin", "TimmBackbone", "Speech2Text2"]
    model_names = [name for name in model_names if name not in names_to_exclude]
    model_names.sort(key=str.lower)

    columns = ["Model", "PyTorch support", "TensorFlow support", "Flax Support"]
    # We'll need widths to properly display everything in the center (+2 is to leave one extra space on each side).

    widths = [len(c) + 2 for c in columns]
    widths[0] = max([len(doc_link) for doc_link in model_name_to_link_mapping.values()]) + 2

    # Build the table per se
    table = "|" + "|".join([_center_text(c, w) for c, w in zip(columns, widths)]) + "|\n"
    # Use ":-----:" format to center-aligned table cell texts
    table += "|" + "|".join([":" + "-" * (w - 2) + ":" for w in widths]) + "|\n"

    check = {True: "✅", False: "❌"}

    for name in model_names:
        if name in MODEL_NAMES_TO_IGNORE:
            continue
        if name in MODEL_NAMES_WITH_SAME_CONFIG.keys():
            prefix = model_name_to_prefix[MODEL_NAMES_WITH_SAME_CONFIG[name]]
        else:
            prefix = model_name_to_prefix[name]
        line = [
            model_name_to_link_mapping[name],
            check[pt_models[prefix]],
            check[tf_models[prefix]],
            check[flax_models[prefix]],
        ]
        table += "|" + "|".join([_center_text(l, w) for l, w in zip(line, widths)]) + "|\n"


SUMMARY:
**Summary:**

`get_model_table_from_auto_modules` function generates an up-to-date model table from the content of auto modules.

**Main Purpose and Functionality:**

* Generates a table with model names, PyTorch, TensorFlow, and Flax support information.
* Retrieves configuration mapping, models, and prefixes from `transformers_module`.
* Flags models for each backend (PT/TF/Flax) using regular expressions.
* Builds the table by iterating through generated model names, checking backend support, and formatting text.

**Notable Implementation Details:**

* Uses dictionaries to store configuration mapping, model names, and prefixes.
* Utilizes regular expressions to flag models for each backend.
* Uses `_center_text` function to format table cell texts.

**Notable Dependencies or Requirements:**

* `transformers_module` module (contains constants and functions).
* `_re_tf_models`, `_re_flax_models`, `_re_pt_models` regular expression patterns (defined in `transformers_module`).


key: chroma:document
string_value:
CODE:

def check_model_table(overwrite=False):
    """
    Check the model table in the index.md is consistent with the state of the lib and potentially fix it.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether or not to overwrite the table when it's not up to date.
    """
    current_table, start_index, end_index, lines = _find_text_in_file(
        filename=os.path.join(PATH_TO_DOCS, "index.md"),
        start_prompt="<!--This table is updated automatically from the auto modules",
        end_prompt="<!-- End table-->",
    )
    new_table = get_model_table_from_auto_modules()

    if current_table != new_table:
        if overwrite:
            with open(os.path.join(PATH_TO_DOCS, "index.md"), "w", encoding="utf-8", newline="\n") as f:
                f.writelines(lines[:start_index] + [new_table] + lines[end_index:])
        else:
            raise ValueError(
                "The model table in the `index.md` has not been updated. Run `make fix-copies` to fix this."


SUMMARY:
**Summary**

This is a Python function named `check_model_table` that checks if the content of the `model table` in the `index.md` file is consistent with its creation via auto-modules. It returns two tables, one previous (`current_table`) and one generated from auto-modules (`new_table`). The primary purpose of this function is to detect inconsistencies and update the `index.md` file accordingly.

**Key Implementation Details**

- This function uses `_find_text_in_file` to extract text ranges containing the model table from `index.md`. 
- It generates a new table using `get_model_table_from_auto_modules`.
- If the tables are different, it either overwrites the outdated table or raises an error with a suggestion to update via `make fix-copies`.

**Notable Dependencies/Requirements**

- `os` (file system operations)
- `PATH_TO_DOCS`, assumed to be a path constant defined elsewhere in the code.
- `_find_text_in_file` and `get_model_table_from_auto_modules` functions, assumed to be implemented elsewhere.


key: chroma:document
string_value:
CODE:

def get_framework(test_class):
    """Infer the framework from the test class `test_class`."""

    if "ModelTesterMixin" in [x.__name__ for x in test_class.__bases__]:
        return "pt"
    elif "TFModelTesterMixin" in [x.__name__ for x in test_class.__bases__]:
        return "tf"
    elif "FlaxModelTesterMixin" in [x.__name__ for x in test_class.__bases__]:
        return "flax"
    else:


SUMMARY:
**Summary:**

`get_framework` function infers the framework associated with a given test class. It searches inheritance hierarchy of `test_class` and returns `"pt"`, `"tf"`, or `"flax"` based on the presence of `ModelTesterMixin` or its subclasses.

**Key Details:**

* Function signature: `def get_framework(test_class)`
* Primary purpose: Infer framework type from test class
* Algorithm:
	+ Search inheritance hierarchy of `test_class`
	+ Check for `ModelTesterMixin`, `TFModelTesterMixin`, or `FlaxModelTesterMixin` subclasses
	+ Return corresponding framework ID ("pt", "tf", or "flax")

**Dependencies/Requirements:**

* None


key: chroma:document
string_value:
CODE:

def get_mapping_for_task(task, framework):
    """Get mappings defined in `XXXPipelineTests` for the task `task`."""
    # Use the cached results
    if PIPELINE_TEST_MAPPING[task].get(framework, None) is not None:
        return PIPELINE_TEST_MAPPING[task][framework]

    pipeline_test_class = pipeline_test_mapping[task]["test"]
    mapping = None

    if framework == "pt":
        mapping = getattr(pipeline_test_class, "model_mapping", None)
    elif framework == "tf":
        mapping = getattr(pipeline_test_class, "tf_model_mapping", None)

    if mapping is not None:
        mapping = dict(mapping.items())

    # cache the results
    PIPELINE_TEST_MAPPING[task][framework] = mapping


SUMMARY:
**Function Summary: `get_mapping_for_task`**

**Purpose:** Returns the task-specific model mapping for a given task and framework, retrieving or caching the result.

**Key Implementation Details:**

* Retrieves cached result if available through `PIPELINE_TEST_MAPPING[task].get(framework, None)`.
* Uses frameworkspecific attributes (`model_mapping`, `tf_model_mapping`) from `pipeline_test_class` to construct the mapping.
* Caches the result in `PIPELINE_TEST_MAPPING[task][framework]`.

**Notable Dependencies and Requirements:**

* `pipeline_test_mapping`: a dictionary containing task-specific pipeline test data.
* `PIPELINE TestingMapping`: an external cache storage for mapped results.

**Method Signatures and Interfaces:**

* `get(framework) -> Mapping dict`


key: chroma:document
string_value:
CODE:

def get_model_for_pipeline_test(test_class, task):
    """Get the model architecture(s) related to the test class `test_class` for a pipeline `task`."""
    framework = get_framework(test_class)
    if framework is None:
        return None
    mapping = get_mapping_for_task(task, framework)
    if mapping is None:
        return None

    config_classes = list({model_class.config_class for model_class in test_class.all_model_classes})
    if len(config_classes) != 1:
        raise ValueError("There should be exactly one configuration class from `test_class.all_model_classes`.")

    # This could be a list/tuple of model classes, but it's rare.
    model_class = mapping.get(config_classes[0], None)
    if isinstance(model_class, (tuple, list)):
        model_class = sorted(model_class, key=lambda x: x.__name__)



SUMMARY:
**Summary:**

The `get_model_for_pipeline_test` function retrieves the model architecture(s) associated with a given test class `test_class` and pipeline task `task`. It achieves this by:

*   Retrieving the framework related to `test_class` using `get_framework`.
*   Resolving the mapping for `task` based on the retrieved framework using `get_mapping_for_task`.
*   Identifying the sole configuration class from `test_class.all_model_classes` and retrieving the corresponding model architecture(s) using this class's configuration.

**Key Details:**

*   The function raises a `ValueError` if more than one configuration class is present in `test_class.all_model_classes`.
*   The model architecture may be returned as a single class, list, or tuple of classes, sorted by their names for consistency.

**Notable Dependencies/Requirements:**

*   `get_framework`
*   `get_mapping_for_task`
*   `test_class` (assuming it has methods like `all_model_classes`).
*   Import statements assume existing models are defined with a `config_class`.


key: chroma:document
string_value:
CODE:

def get_pipeline_model_mapping(test_class):
    """Get `pipeline_model_mapping` for `test_class`."""
    mapping = [(task, get_model_for_pipeline_test(test_class, task)) for task in pipeline_test_mapping]
    mapping = sorted([(task, model) for task, model in mapping if model is not None], key=lambda x: x[0])



SUMMARY:
**Function Summary**

`get_pipeline_model_mapping` returns a dictionary mapping test classes to pipelines, where each pipeline is executed and mapped to its corresponding model. 

**Key Details**

* Iterates through `pipeline_test_mapping` using `test_class` as input.
* Applies `get_model_for_pipeline_test` function for each task.
* Filters out None values and sorts the results by task name.

**Return Value**

A dictionary with task names as keys and pipeline models as values. 

**Dependences/Requirements**

* `pipeline_test_mapping`: A list of tuples containing test class names and corresponding pipeline tasks.
* `get_model_for_pipeline_test(test_class, task)`: A function that returns a model for a given pipeline task.


key: chroma:document
string_value:
CODE:

def get_pipeline_model_mapping_string(test_class):
    """Get `pipeline_model_mapping` for `test_class` as a string (to be added to the test file).

    This will be a 1-line string. After this is added to a test file, `make style` will format it beautifully.
    """
    framework = get_framework(test_class)
    if framework == "pt":
        framework = "torch"
    default_value = "{}"

    mapping = get_pipeline_model_mapping(test_class)
    if len(mapping) == 0:
        return ""

    texts = []
    for task, model_classes in mapping.items():
        if isinstance(model_classes, (tuple, list)):
            # A list/tuple of model classes
            value = "(" + ", ".join([x.__name__ for x in model_classes]) + ")"
        else:
            # A single model class
            value = model_classes.__name__
        texts.append(f'"{task}": {value}')
    text = "{" + ", ".join(texts) + "}"
    text = f"pipeline_model_mapping = {text} if is_{framework}_available() else {default_value}"



SUMMARY:
**Function Summary: `get_pipeline_model_mapping_string`**

**Purpose and Functionality:** This function generates a 1-line string representing the pipeline model mapping for a given test class, which can be added to a test file.

**Key Implementation Details:**

* Uses the `get_framework` function to determine the framework type ("pt" or "torch") based on the input test class.
* Calls `get_pipeline_model_mapping` to retrieve the pipeline model mapping.
* Constructs a list of text strings (`"task": value`) representing each model class/class tuple pair in the mapping.
* Joins these strings into a single JSON-like string and formats it with conditional logic using the `is_{framework}_available()` function.

**Notable Dependencies and Requirements:**

* The `get_framework` function
* The `get_pipeline_model_mapping` function
* Conditional availability check using `is_{framework}_available()`
* Python dictionary-like formatting (`{key: value}`) for JSON-style strings.


key: chroma:document
string_value:
CODE:

def is_valid_test_class(test_class):
    """Restrict to `XXXModelTesterMixin` and should be a subclass of `unittest.TestCase`."""
    base_class_names = {"ModelTesterMixin", "TFModelTesterMixin", "FlaxModelTesterMixin"}
    if not issubclass(test_class, unittest.TestCase):
        return False


SUMMARY:
**Summary**

Validates if a given test class inherits from `unittest.TestCase` and is derived from one of the specified model tester mixins:
```python
def is_valid_test_class(test_class):
    """Restricts to `XXXModelTesterMixin` and should be a subclass of `unittest.TestCase`.
    
    Parameters:
        test_class (class): The class to validate.
    
    Returns:
        bool: True if valid, False otherwise.
    """
    # Constants
    base ClassNames = {"ModelTesterMixin", "TFModelTesterMixin", "FlaxModelTesterMixin"}
    
    # Validation check
    def isValidBaseClass(base_class):
        return len(base_classNames.intersection([x.__name__ for x in base_class.__bases__])) > 0
    
    # Check if the test class is a subclass of TestCase and has one of the model tester mixins as an base class
    return not issubclass(test_class, unittest.TestCase) or isValidBaseClass(test_class)
```
**Key Details**

* `unittest.TestCase` class validation
* `ModelTesterMixin`, `TFModelTesterMixin`, and `FlaxModelTesterMixin` derived classes are expected

Note: The `is_valid_test_class` function returns False if the test class does not inherit from `unittest_TestCase` or if it does not have a mixin as an base class, whereas the original code would return False immediately without checking for any mixin.


key: chroma:document
string_value:
CODE:

def find_test_class(test_file):
    """Find a test class in `test_file` to which we will add `pipeline_model_mapping`."""
    test_classes = [x for x in get_test_classes(test_file) if is_valid_test_class(x)]

    target_test_class = None
    for test_class in test_classes:
        # If a test class has defined `pipeline_model_mapping`, let's take it
        if getattr(test_class, "pipeline_model_mapping", None) is not None:
            target_test_class = test_class
            break
    # Take the test class with the shortest name (just a heuristic)
    if target_test_class is None and len(test_classes) > 0:
        target_test_class = sorted(test_classes, key=lambda x: (len(x.__name__), x.__name__))[0]



SUMMARY:
**Function Summary:**

`find_test_class(test_file)` finds and selects the test class from the `test_file` to which `pipeline_model_mapping` will be added.

**Key Implementation Details:**

* Iterates over valid test classes in `test_file`.
* Checks for presence of `pipeline_model_mapping` attribute; selects first matching class.
* Fallbacks to shortest class name heuristic if no match found.

**Notable Dependencies and Requirements:**

* `get_test_classes(test_file)`: returns list of test classes in `test_file`.
* `is_valid_test_class(x)`: checks validity of a test class.
* External dependency: `test_file` (required).


key: chroma:document
string_value:
CODE:

def find_block_ending(lines, start_idx, indent_level):
    end_idx = start_idx
    for idx, line in enumerate(lines[start_idx:]):
        indent = len(line) - len(line.lstrip())
        if idx == 0 or indent > indent_level or (indent == indent_level and line.strip() == ")"):
            end_idx = start_idx + idx
        elif idx > 0 and indent <= indent_level:
            # Outside the definition block of `pipeline_model_mapping`
            break



SUMMARY:
**Summary:**

`find_block_ending`: Finds the index of the last line in a code block, ending with a matching bracket.

**Purpose:** The function locates the closing bracket of a block defined by indentation levels.

**Key Implementation Details:**

* Iterates through lines starting from `start_idx`.
* Checks for matching bracket using indentation level.
* Returns the index of the block's end (`end_idx`).

**Note:** The function ensures the block does not exceed a line containing the string ")" while also considering lines outside the block defined by "pipeline_model_mapping".


key: chroma:document
string_value:
CODE:

def add_pipeline_model_mapping(test_class, overwrite=False):
    """Add `pipeline_model_mapping` to `test_class`."""
    if getattr(test_class, "pipeline_model_mapping", None) is not None:
        if not overwrite:
            return "", -1

    line_to_add = get_pipeline_model_mapping_string(test_class)
    if len(line_to_add) == 0:
        return "", -1
    line_to_add = line_to_add + "\n"

    # The code defined the class `test_class`
    class_lines, class_start_line_no = inspect.getsourcelines(test_class)
    # `inspect` gives the code for an object, including decorator(s) if any.
    # We (only) need the exact line of the class definition.
    for idx, line in enumerate(class_lines):
        if line.lstrip().startswith("class "):
            class_lines = class_lines[idx:]
            class_start_line_no += idx
            break
    class_end_line_no = class_start_line_no + len(class_lines) - 1

    # The index in `class_lines` that starts the definition of `all_model_classes`, `all_generative_model_classes` or
    # `pipeline_model_mapping`. This assumes they are defined in such order, and we take the start index of the last
    # block that appears in a `test_class`.
    start_idx = None
    # The indent level of the line at `class_lines[start_idx]` (if defined)
    indent_level = 0
    # To record if `pipeline_model_mapping` is found in `test_class`.
    def_line = None
    for idx, line in enumerate(class_lines):
        if line.strip().startswith("all_model_classes = "):
            indent_level = len(line) - len(line.lstrip())
            start_idx = idx
        elif line.strip().startswith("all_generative_model_classes = "):
            indent_level = len(line) - len(line.lstrip())
            start_idx = idx
        elif line.strip().startswith("pipeline_model_mapping = "):
            indent_level = len(line) - len(line.lstrip())
            start_idx = idx
            def_line = line
            break

    if start_idx is None:
        return "", -1
    # Find the ending index (inclusive) of the above found block.
    end_idx = find_block_ending(class_lines, start_idx, indent_level)

    # Extract `is_xxx_available()` from existing blocks: some models require specific libraries like `timm` and use
    # `is_timm_available()` instead of `is_torch_available()`.
    # Keep leading and trailing whitespaces
    r = re.compile(r"\s(is_\S+?_available\(\))\s")
    for line in class_lines[start_idx : end_idx + 1]:
        backend_condition = r.search(line)
        if backend_condition is not None:
            # replace the leading and trailing whitespaces to the space character " ".
            target = " " + backend_condition[0][1:-1] + " "
            line_to_add = r.sub(target, line_to_add)
            break

    if def_line is None:
        # `pipeline_model_mapping` is not defined. The target index is set to the ending index (inclusive) of
        # `all_model_classes` or `all_generative_model_classes`.
        target_idx = end_idx
    else:
        # `pipeline_model_mapping` is defined. The target index is set to be one **BEFORE** its start index.
        target_idx = start_idx - 1
        # mark the lines of the currently existing `pipeline_model_mapping` to be removed.
        for idx in range(start_idx, end_idx + 1):
            # These lines are going to be removed before writing to the test file.
            class_lines[idx] = None  # noqa

    # Make sure the test class is a subclass of `PipelineTesterMixin`.
    parent_classes = [x.__name__ for x in test_class.__bases__]
    if "PipelineTesterMixin" not in parent_classes:
        # Put `PipelineTesterMixin` just before `unittest.TestCase`
        _parent_classes = [x for x in parent_classes if x != "TestCase"] + ["PipelineTesterMixin"]
        if "TestCase" in parent_classes:
            # Here we **assume** the original string is always with `unittest.TestCase`.
            _parent_classes.append("unittest.TestCase")
        parent_classes = ", ".join(_parent_classes)
        for idx, line in enumerate(class_lines):
            # Find the ending of the declaration of `test_class`
            if line.strip().endswith("):"):
                # mark the lines of the declaration of `test_class` to be removed
                for _idx in range(idx + 1):
                    class_lines[_idx] = None  # noqa
                break
        # Add the new, one-line, class declaration for `test_class`
        class_lines[0] = f"class {test_class.__name__}({parent_classes}):\n"

    # Add indentation
    line_to_add = " " * indent_level + line_to_add
    # Insert `pipeline_model_mapping` to `class_lines`.
    # (The line at `target_idx` should be kept by definition!)
    class_lines = class_lines[: target_idx + 1] + [line_to_add] + class_lines[target_idx + 1 :]
    # Remove the lines that are marked to be removed
    class_lines = [x for x in class_lines if x is not None]

    # Move from test class to module (in order to write to the test file)
    module_lines = inspect.getsourcelines(inspect.getmodule(test_class))[0]
    # Be careful with the 1-off between line numbers and array indices
    module_lines = module_lines[: class_start_line_no - 1] + class_lines + module_lines[class_end_line_no:]
    code = "".join(module_lines)

    moddule_file = inspect.getsourcefile(test_class)
    with open(moddule_file, "w", encoding="UTF-8", newline="\n") as fp:
        fp.write(code)



SUMMARY:
Here is a concise summary of the provided function code chunk:

**Function Purpose:**
The `add_pipeline_model_mapping` function adds or ovewrites the `pipeline_model_mapping` attribute to a given test class (`test_class`).

**Key Implementation Details:**

1.  The function checks if `test_class` already has a defined `pipeline_model_mapping` and only overwrites it if set to `False`.
2.  It searches for the line that defines the previous block (e.g., `all_model_classes`, `all_generative_model_classes`, `pipeline_model_mapping`) to determine the ending index of the block.
3.  If a matching line is found, the function removes any leading whitespaces and replaces it with a specific regex pattern (`is_\S+_(available)\s`) before extracting the new line of code for `pipeline_model_mapping`.
4.  It ensures that the modified test class name is appended after its parent classes using the `"unittest.TestCase"`
5.  After adding or overwriting `pipeline_model_mapping`, it removes the lines marked to be removed.

**Dependencies and Requirements:**

1.  Uses the `inspect` module for code inspection.
2.  References related classes/functions by exact name, such as getting class lines using its sourcelines method.
3.  Requires external dependencies like regular expression (`re`) library.

**Return Value:** The added or overwritten line of new `pipeline_model_mapping`.


key: chroma:document
string_value:
CODE:

def add_pipeline_model_mapping_to_test_file(test_file, overwrite=False):
    """Add `pipeline_model_mapping` to `test_file`."""
    test_class = find_test_class(test_file)
    if test_class:


SUMMARY:
**Summary:**

`add_pipeline_model_mapping_to_test_file`: Adds the `pipeline_model_mapping` configuration to a `test_file`, overwriting if specified.

**Key Details:**

- Calls `find_test_class` with `test_file` as input.
- If a test class is found, it calls `add_pipeline_model_mapping` with the class and an optional overwrite flag.
- The primary purpose of using `overwrite=False` in `add_pipeline_model_mapping` if `test_class` exists.

 Relevant methods: 
 - `find_test_class(test_file)`
 - `add_pipeline_model_mapping(test_class, overwrite)`

Note:
- External dependency: `find_test_class` and `add_pipeline_model_mapping` functions are assumed to be defined elsewhere.


key: chroma:document
string_value:
CODE:

def get_processor_types_from_config_class(config_class, allowed_mappings=None):
    """Return a tuple of processors for `config_class`.

    We use `tuple` here to include (potentially) both slow & fast tokenizers.
    """

    # To make a uniform return type
    def _to_tuple(x):
        if not isinstance(x, collections.abc.Sequence):
            x = (x,)
        else:
            x = tuple(x)
        return x

    if allowed_mappings is None:
        allowed_mappings = ["processor", "tokenizer", "image_processor", "feature_extractor"]

    processor_types = ()

    # Check first if a model has `ProcessorMixin`. Otherwise, check if it has tokenizers, and/or an image processor or
    # a feature extractor
    if config_class in PROCESSOR_MAPPING and "processor" in allowed_mappings:
        processor_types = _to_tuple(PROCESSOR_MAPPING[config_class])
    else:
        if config_class in TOKENIZER_MAPPING and "tokenizer" in allowed_mappings:
            processor_types = TOKENIZER_MAPPING[config_class]

        if config_class in IMAGE_PROCESSOR_MAPPING and "image_processor" in allowed_mappings:
            processor_types += _to_tuple(IMAGE_PROCESSOR_MAPPING[config_class])
        elif config_class in FEATURE_EXTRACTOR_MAPPING and "feature_extractor" in allowed_mappings:
            processor_types += _to_tuple(FEATURE_EXTRACTOR_MAPPING[config_class])

    # Remark: some configurations have no processor at all. For example, generic composite models like
    # `EncoderDecoderModel` is used for any (compatible) text models. Also, `DecisionTransformer` doesn't
    # require any processor.

    # We might get `None` for some tokenizers - remove them here.
    processor_types = tuple(p for p in processor_types if p is not None)



SUMMARY:
**Code Summary**

```
get_processor_types_from_config_class(config_class, allowed_mappings=None)
-------------------------

Purpose:
Extracts and returns a tuple of supported processors or tokenizer types based on the provided config class.

Functionality:

* Takes a config class and an optional list of allowed mappings as input.
* Checks if the config class has a direct mapping in PROCESSOR_MAPPING, then uses that.
* If not, checks for indirect mappings through TOKENIZER_MAPPING, IMAGE_PROCESSOR_MAPPING, or FEATURE_EXTRACTOR_MAPPING.
* Removes None values from the resulting processor types tuple.

Key Implementation Details:

* Uses `_to_tuple` function to ensure uniform return type (sequence or scalar).
* Checks for specific allowed mappings and their corresponding mappings in PROCESSOR_MAPPING and other alias dictionaries.
* Uses tuple concatenation to combine multiple processor types into a single tuple.

Dependencies/Requirements:

* PROCESSOR_MAPPING, TOKENIZER_MAPPING, IMAGE_PROCESSOR_MAPPING, FEATURE_EXTRACTOR_MAPPING
* collections.abc.Sequence (for isinstance check)
```

Note: The code summary is written in a concise and focused manner, highlighting the main purpose, functionality, key implementation details, and dependencies.


key: chroma:document
string_value:
CODE:

def get_architectures_from_config_class(config_class, arch_mappings, models_to_skip=None):
    """Return a tuple of all possible architectures attributed to a configuration class `config_class`.

    For example, BertConfig -> [BertModel, BertForMaskedLM, ..., BertForQuestionAnswering].
    """
    # A model architecture could appear in several mappings. For example, `BartForConditionalGeneration` is in
    #   - MODEL_FOR_PRETRAINING_MAPPING_NAMES
    #   - MODEL_WITH_LM_HEAD_MAPPING_NAMES
    #   - MODEL_FOR_MASKED_LM_MAPPING_NAMES
    #   - MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES
    # We avoid the duplication.
    architectures = set()

    if models_to_skip is None:
        models_to_skip = []
    models_to_skip = UNCONVERTIBLE_MODEL_ARCHITECTURES.union(models_to_skip)

    for mapping in arch_mappings:
        if config_class in mapping:
            models = mapping[config_class]
            models = tuple(models) if isinstance(models, collections.abc.Sequence) else (models,)
            for model in models:
                if model.__name__ not in models_to_skip:
                    architectures.add(model)

    architectures = tuple(architectures)



SUMMARY:
**Code Summary:**

### Purpose and Functionality

`get_architectures_from_config_class` returns a tuple of possible architectures attributed to a configuration class `config_class`, filtering out excluded models using provided mappings.

### Key Implementation Details

*   Uses a set to avoid duplications across multiple mapping sources.
*   Filters out models based on a boolean `models_to_skip`.

### Notable Dependencies or Requirements

*   `arch_mappings`: dictionary of configuration classes mapped to architectures
*   `models_to_skip`: optional list of excluded model names (defaults to empty set)
*   `UNCONVERTIBLE_MODEL_ARCHITECTURES`: pre-defined set of inconvertible architecture models


key: chroma:document
string_value:
CODE:

def get_config_class_from_processor_class(processor_class):
    """Get the config class from a processor class.

    Some config/model classes use tokenizers/feature_extractors from other models. For example, `GPT-J` uses
    `GPT2Tokenizer`. If no checkpoint is found for a config class, or a checkpoint is found without necessary file(s) to
    create the processor for `processor_class`, we get the config class that corresponds to `processor_class` and use it
    to find a checkpoint in order to create the processor.
    """

    processor_prefix = processor_class.__name__
    for postfix in ["TokenizerFast", "Tokenizer", "ImageProcessor", "FeatureExtractor", "Processor"]:
        processor_prefix = processor_prefix.replace(postfix, "")

    # `Wav2Vec2CTCTokenizer` -> `Wav2Vec2Config`
    if processor_prefix == "Wav2Vec2CTC":
        processor_prefix = "Wav2Vec2"

    # Find the new configuration class
    new_config_name = f"{processor_prefix}Config"
    new_config_class = getattr(transformers_module, new_config_name)



SUMMARY:
**Summary:**

This function, `get_config_class_from_processor_class`, retrieves the corresponding config class from a given processor class. It checks for prefixes in the processor class name to determine the necessary config class, and returns it if a matching checkpoint is found.

**Key Implementation Details:**

* Replaced the original prefix with a shortened version (e.g., "Tokenizer" -> "")
* Handles specific cases for Wav2Vec2CTC and other models
* Uses the `getattr` function to dynamically retrieve the config class from the `transformers_module`

**Notable Dependencies and Requirements:**

* `transformers_module`: assumed to be a module with config classes registered under their name (e.g., `Wav2Vec2Config`, `GPT2Tokenizer`)
* Tokenizer/feature extractor dependencies specific to each model are not explicitly handled; may require additional configuration or file presence checks.


key: chroma:document
string_value:
CODE:

def build_processor(config_class, processor_class, allow_no_checkpoint=False):
    """Create a processor for `processor_class`.

    If a processor is not able to be built with the original arguments, this method tries to change the arguments and
    call itself recursively, by inferring a new `config_class` or a new `processor_class` from another one, in order to
    find a checkpoint containing the necessary files to build a processor.

    The processor is not saved here. Instead, it will be saved in `convert_processors` after further changes in
    `convert_processors`. For each model architecture`, a copy will be created and saved along the built model.
    """
    # Currently, this solely uses the docstring in the source file of `config_class` to find a checkpoint.
    checkpoint = get_checkpoint_from_config_class(config_class)

    if checkpoint is None:
        # try to get the checkpoint from the config class for `processor_class`.
        # This helps cases like `XCLIPConfig` and `VideoMAEFeatureExtractor` to find a checkpoint from `VideoMAEConfig`.
        config_class_from_processor_class = get_config_class_from_processor_class(processor_class)
        checkpoint = get_checkpoint_from_config_class(config_class_from_processor_class)

    processor = None
    try:
        processor = processor_class.from_pretrained(checkpoint)
    except Exception as e:
        logger.error(f"{e.__class__.__name__}: {e}")

    # Try to get a new processor class from checkpoint. This is helpful for a checkpoint without necessary file to load
    # processor while `processor_class` is an Auto class. For example, `sew` has `Wav2Vec2Processor` in
    # `PROCESSOR_MAPPING_NAMES`, its `tokenizer_class` is `AutoTokenizer`, and the checkpoint
    # `https://huggingface.co/asapp/sew-tiny-100k` has no tokenizer file, but we can get
    # `tokenizer_class: Wav2Vec2CTCTokenizer` from the config file. (The new processor class won't be able to load from
    # `checkpoint`, but it helps this recursive method to find a way to build a processor).
    if (
        processor is None
        and checkpoint is not None
        and issubclass(processor_class, (PreTrainedTokenizerBase, AutoTokenizer))
    ):
        try:
            config = AutoConfig.from_pretrained(checkpoint)
        except Exception as e:
            logger.error(f"{e.__class__.__name__}: {e}")
            config = None
        if config is not None:
            if not isinstance(config, config_class):
                raise ValueError(
                    f"`config` (which is of type {config.__class__.__name__}) should be an instance of `config_class`"
                    f" ({config_class.__name__})!"
                )
            tokenizer_class = config.tokenizer_class
            new_processor_class = None
            if tokenizer_class is not None:
                new_processor_class = getattr(transformers_module, tokenizer_class)
                if new_processor_class != processor_class:
                    processor = build_processor(config_class, new_processor_class)
            # If `tokenizer_class` is not specified in `config`, let's use `config` to get the process class via auto
            # mappings, but only allow the tokenizer mapping being used. This is to make `Wav2Vec2Conformer` build
            if processor is None:
                new_processor_classes = get_processor_types_from_config_class(
                    config.__class__, allowed_mappings=["tokenizer"]
                )
                # Used to avoid infinite recursion between a pair of fast/slow tokenizer types
                names = [
                    x.__name__.replace("Fast", "") for x in [processor_class, new_processor_class] if x is not None
                ]
                new_processor_classes = [
                    x for x in new_processor_classes if x is not None and x.__name__.replace("Fast", "") not in names
                ]
                if len(new_processor_classes) > 0:
                    new_processor_class = new_processor_classes[0]
                    # Let's use fast tokenizer if there is any
                    for x in new_processor_classes:
                        if x.__name__.endswith("Fast"):
                            new_processor_class = x
                            break
                    processor = build_processor(config_class, new_processor_class)

    if processor is None:
        # Try to build each component (tokenizer & feature extractor) of a `ProcessorMixin`.
        if issubclass(processor_class, ProcessorMixin):
            attrs = {}
            for attr_name in processor_class.attributes:
                attrs[attr_name] = []
                # This could be a tuple (for tokenizers). For example, `CLIPProcessor` has
                #   - feature_extractor_class = "CLIPFeatureExtractor"
                #   - tokenizer_class = ("CLIPTokenizer", "CLIPTokenizerFast")
                attr_class_names = getattr(processor_class, f"{attr_name}_class")
                if not isinstance(attr_class_names, tuple):
                    attr_class_names = (attr_class_names,)

                for name in attr_class_names:
                    attr_class = getattr(transformers_module, name)
                    attr = build_processor(config_class, attr_class)
                    if attr is not None:
                        attrs[attr_name].append(attr)

            # try to build a `ProcessorMixin`, so we can return a single value
            if all(len(v) > 0 for v in attrs.values()):
                try:
                    processor = processor_class(**{k: v[0] for k, v in attrs.items()})
                except Exception as e:
                    logger.error(f"{e.__class__.__name__}: {e}")
        else:
            # `checkpoint` might lack some file(s) to load a processor. For example, `facebook/hubert-base-ls960`
            # has no tokenizer file to load `Wav2Vec2CTCTokenizer`. In this case, we try to build a processor
            # with the configuration class (for example, `Wav2Vec2Config`) corresponding to `processor_class`.
            config_class_from_processor_class = get_config_class_from_processor_class(processor_class)
            if config_class_from_processor_class != config_class:
                processor = build_processor(config_class_from_processor_class, processor_class)

    # Try to create an image processor or a feature extractor without any checkpoint
    if (
        processor is None
        and allow_no_checkpoint
        and (issubclass(processor_class, BaseImageProcessor) or issubclass(processor_class, FeatureExtractionMixin))
    ):
        try:
            processor = processor_class()
        except Exception as e:
            logger.error(f"{e.__class__.__name__}: {e}")

    # validation
    if processor is not None:
        if not (isinstance(processor, processor_class) or processor_class.__name__.startswith("Auto")):
            raise ValueError(
                f"`processor` (which is of type {processor.__class__.__name__}) should be an instance of"
                f" {processor_class.__name__} or an Auto class!"
            )



SUMMARY:
**Function Summary: `build_processor`**

**Purpose:** Create a processor for the specified `processor_class` using a configuration class from `config_class`.

**Main Functionality:**

1. Use `get_checkpoint_from_config_class` to find a checkpoint in the source file of `config_class`.
2. If no checkpoint is found, attempt to get one from `processore_class`.
3. Try to create an instance of `processor_class` from the checkpoint.
4. If creation fails, use recursive approach to try alternative configurations.

**Key Implementation Details:**

*   Uses `get_config_class_from_processor_class` to map `processor_class` to a configuration class.
*   Tries to get a new processor class using AutoMapper or by extracting tokenizer classes from the config file.
*   Creates an image processor or feature extractor without a checkpoint if `allow_no_checkpoint` is set.

**Notable Dependencies and Requirements:**

*   Uses `transformers_module` for loading processor classes.
*   References related classes/functions like `get_config_class_from_processor_class`, `pretrained`, `logger`.
*   Assumes the existence of certain configuration files (e.g., `processorer_mapping_names`).


key: chroma:document
string_value:
CODE:

def get_tiny_config(config_class, model_class=None, **model_tester_kwargs):
    """Retrieve a tiny configuration from `config_class` using each model's `ModelTester`.

    Args:
        config_class: Subclass of `PreTrainedConfig`.

    Returns:
        An instance of `config_class` with tiny hyperparameters
    """
    model_type = config_class.model_type

    # For model type like `data2vec-vision` and `donut-swin`, we can't get the config/model file name directly via
    # `model_type` as it would be sth. like `configuration_data2vec_vision.py`.
    # A simple way is to use `inspect.getsourcefile(config_class)`.
    config_source_file = inspect.getsourcefile(config_class)
    # The modeling file name without prefix (`modeling_`) and postfix (`.py`)
    modeling_name = config_source_file.split(os.path.sep)[-1].replace("configuration_", "").replace(".py", "")

    try:
        print("Importing", model_type_to_module_name(model_type))
        module_name = model_type_to_module_name(model_type)
        if not modeling_name.startswith(module_name):
            raise ValueError(f"{modeling_name} doesn't start with {module_name}!")
        test_file = os.path.join("tests", "models", module_name, f"test_modeling_{modeling_name}.py")
        models_to_model_testers = get_model_to_tester_mapping(test_file)
        # Find the model tester class
        model_tester_class = None
        tester_classes = []
        if model_class is not None:
            tester_classes = get_tester_classes_for_model(test_file, model_class)
        else:
            for _tester_classes in models_to_model_testers.values():
                tester_classes.extend(_tester_classes)
        if len(tester_classes) > 0:
            # sort with the length of the class names first, then the alphabetical order
            # This is to avoid `T5EncoderOnlyModelTest` is used instead of `T5ModelTest`, which has
            # `is_encoder_decoder=False` and causes some pipeline tests failing (also failures in `Optimum` CI).
            # TODO: More fine grained control of the desired tester class.
            model_tester_class = sorted(tester_classes, key=lambda x: (len(x.__name__), x.__name__))[0]
    except ModuleNotFoundError:
        error = f"Tiny config not created for {model_type} - cannot find the testing module from the model name."
        raise ValueError(error)

    if model_tester_class is None:
        error = f"Tiny config not created for {model_type} - no model tester is found in the testing module."
        raise ValueError(error)

    # CLIP-like models have `text_model_tester` and `vision_model_tester`, and we need to pass `vocab_size` to
    # `text_model_tester` via `text_kwargs`. The same trick is also necessary for `Flava`.

    if "vocab_size" in model_tester_kwargs:
        if "text_kwargs" in inspect.signature(model_tester_class.__init__).parameters.keys():
            vocab_size = model_tester_kwargs.pop("vocab_size")
            model_tester_kwargs["text_kwargs"] = {"vocab_size": vocab_size}

    # `parent` is an instance of `unittest.TestCase`, but we don't need it here.
    model_tester = model_tester_class(parent=None, **model_tester_kwargs)

    if hasattr(model_tester, "get_pipeline_config"):
        config = model_tester.get_pipeline_config()
    elif hasattr(model_tester, "prepare_config_and_inputs"):
        # `PoolFormer` has no `get_config` defined. Furthermore, it's better to use `prepare_config_and_inputs` even if
        # `get_config` is defined, since there might be some extra changes in `prepare_config_and_inputs`.
        config = model_tester.prepare_config_and_inputs()[0]
    elif hasattr(model_tester, "get_config"):
        config = model_tester.get_config()
    else:
        error = (
            f"Tiny config not created for {model_type} - the model tester {model_tester_class.__name__} lacks"
            " necessary method to create config."
        )
        raise ValueError(error)

    # make sure this is long enough (some model tester has `20` for this attr.) to pass `text-generation`
    # pipeline tests.
    max_positions = []
    for key in ["max_position_embeddings", "max_source_positions", "max_target_positions"]:
        if getattr(config, key, 0) > 0:
            max_positions.append(getattr(config, key))
        if getattr(config, "text_config", None) is not None:
            if getattr(config.text_config, key, None) is not None:
                max_positions.append(getattr(config.text_config, key))
    if len(max_positions) > 0:
        max_position = max(200, min(max_positions))
        for key in ["max_position_embeddings", "max_source_positions", "max_target_positions"]:
            if getattr(config, key, 0) > 0:
                setattr(config, key, max_position)
            if getattr(config, "text_config", None) is not None:
                if getattr(config.text_config, key, None) is not None:
                    setattr(config.text_config, key, max_position)



SUMMARY:
**Summary:**

`get_tiny_config` function retrieves a tiny configuration from `config_class` using each model's `ModelTester`. 

**Main Purpose and Functionality:**

* The function takes a `config_class` (subclass of `PreTrainedConfig`) as input.
* It retrieves the corresponding `model_tester_class` from the testing module based on the `model_type`.
* It creates an instance of the `model_tester_class` with updated arguments to set `vocab_size`.
* It creates a configuration using the model tester's `get_pipeline_config`, `prepare_config_and_inputs`, or `get_config` method.
* If necessary, it sets max positions in the configuration.

**Key Implementation Details:**

* Uses dynamic imports and path manipulation to retrieve the testing module based on `model_type`.
* Sets vocab size for models with a language tokenizer.
* Uses various methods from `model_tester_class` (e.g., `get_pipeline_config`, `prepare_config_and_inputs`) to create the configuration.

**Notable Dependencies or Requirements:**

* `config_class` (subclass of `PreTrainedConfig`)
* `model_type` (determined by `config_class.model_type`)
* Various model testers and corresponding modules
* Testing framework (e.g., `unittest`)


key: chroma:document
string_value:
CODE:

def convert_tokenizer(tokenizer_fast: PreTrainedTokenizerFast):
    new_tokenizer = tokenizer_fast.train_new_from_iterator(
        data["training_ds"]["text"], TARGET_VOCAB_SIZE, show_progress=False
    )

    # Make sure it at least runs
    if not isinstance(new_tokenizer, LayoutLMv3TokenizerFast):
        new_tokenizer(data["testing_ds"]["text"])



SUMMARY:
**Concise Summary**

This function creates or updates a `LayoutLMv3TokenizerFast` instance using the provided `PreTrainedTokenizerFast`. It:

* Trains a new tokenizer from the training data with a specified vocabulary size (`TARGET_VOCAB_SIZE`)
* Checks if the created tokenizer is of type `LayoutLMv3TokenizerFast`, and if not, uses the testing data to create an alternative tokenizer
* Returns the newly created or updated `LayoutLMv3TokenizerFast` instance.

**Details:**

* Main purpose: tokenization for a specific text dataset using `LayoutLMv3TokenizerFast`.
* Key implementation details:
	+ Tokenizer training and update.
	+ Type checking for `LayoutLMv3TokenizerFast`.
* Notable dependencies:
	+ `PreTrainedTokenizerFast` from Hugging Face library.
	+ `data["training_ds"]["text"]` and `data["testing_ds"]["text"]` datasets.


key: chroma:document
string_value:
CODE:

def convert_feature_extractor(feature_extractor, tiny_config):
    to_convert = False
    kwargs = {}
    if hasattr(tiny_config, "image_size"):
        kwargs["size"] = tiny_config.image_size
        kwargs["crop_size"] = tiny_config.image_size
        to_convert = True
    elif (
        hasattr(tiny_config, "vision_config")
        and tiny_config.vision_config is not None
        and hasattr(tiny_config.vision_config, "image_size")
    ):
        kwargs["size"] = tiny_config.vision_config.image_size
        kwargs["crop_size"] = tiny_config.vision_config.image_size
        to_convert = True

    # Speech2TextModel specific.
    if hasattr(tiny_config, "input_feat_per_channel"):
        kwargs["feature_size"] = tiny_config.input_feat_per_channel
        kwargs["num_mel_bins"] = tiny_config.input_feat_per_channel
        to_convert = True

    if to_convert:
        feature_extractor = feature_extractor.__class__(**kwargs)

    # Sanity check: on tiny image feature extractors, a large image size results in slow CI -- up to the point where it
    # can result in timeout issues.
    if (
        isinstance(feature_extractor, BaseImageProcessor)
        and hasattr(feature_extractor, "size")
        and isinstance(feature_extractor.size, dict)
    ):
        largest_image_size = max(feature_extractor.size.values())
        if largest_image_size > 64:
            # hardcoded exceptions
            models_with_large_image_size = ("deformable_detr", "flava", "grounding_dino", "mgp_str", "swiftformer")
            if any(model_name in tiny_config.model_type for model_name in models_with_large_image_size):
                pass
            else:
                raise ValueError(
                    f"Image size of {tiny_config.model_type} is too large ({feature_extractor.size}). "
                    "Please reduce it to 64 or less on each dimension. The following steps are usually the "
                    "easiest solution: 1) confirm that you're setting `image_size` in your ModelTester class; "
                    "2) ensure that it gets passed to the tester config init, `get_config()`."
                )



SUMMARY:
**Function Summary: Convert Feature Extractor**

Converts a `feature_extractor` instance to a new configuration based on `tiny_config`.

**Main Purpose and Functionality:**
The function `convert_feature_extractor` takes two parameters, `feature_extractor` and `tiny_config`, and converts the former to fit the latter's specifications. It checks for various attributes in `tiny_config` (e.g., `image_size`, `vision_config.image_size`) and updates the `kwargs` dictionary accordingly.

**Key Implementation Details:**

* Updates `kwargs["size"]` and `kwargs["crop_size"]` based on `image_size` or `vision_config.image_size`
* Sets `feature_size` and `num_mel_bins` for `Speech2TextModel`
* Checks for large image sizes in BaseImageProcessor instances (up to 64 pixels)
* Raises a ValueError if an incompatible model type is used with too large an image size

**Notable Dependencies/Requirements:**

* `BaseImageProcessor`, `tiny_config`, and related classes/functions are imported but not listed explicitly
* External dependencies on ModelTester, get_config(), ImageSize, VisionConfig are implied but not specified


key: chroma:document
string_value:
CODE:

def convert_processors(processors, tiny_config, output_folder, result):
    """Change a processor to work with smaller inputs.

    For tokenizers, we try to reduce their vocabulary size.

    For feature extractor, we use smaller image size or change
    other attributes using the values from `tiny_config`. See `convert_feature_extractor`.

    This method should not fail: we catch the errors and put them in `result["warnings"]` with descriptive messages.
    """

    def _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=False):
        """Set tokenizer(s) to `None` if the fast/slow tokenizers have different values for `vocab_size` or `length`.

        If `keep_fast_tokenizer=True`, the fast tokenizer will be kept.
        """
        # sanity check 1: fast and slow tokenizers should be compatible (vocab_size)
        if fast_tokenizer is not None and slow_tokenizer is not None:
            if fast_tokenizer.vocab_size != slow_tokenizer.vocab_size:
                warning_message = (
                    "The fast/slow tokenizers "
                    f"({fast_tokenizer.__class__.__name__}/{slow_tokenizer.__class__.__name__}) have different "
                    "vocabulary size: "
                    f"fast_tokenizer.vocab_size = {fast_tokenizer.vocab_size} and "
                    f"slow_tokenizer.vocab_size = {slow_tokenizer.vocab_size}."
                )
                result["warnings"].append(warning_message)
                if not keep_fast_tokenizer:
                    fast_tokenizer = None
                slow_tokenizer = None

        # sanity check 2: fast and slow tokenizers should be compatible (length)
        if fast_tokenizer is not None and slow_tokenizer is not None:
            if len(fast_tokenizer) != len(slow_tokenizer):
                warning_message = (
                    f"The fast/slow tokenizers () have different length: "
                    f"len(fast_tokenizer) = {len(fast_tokenizer)} and "
                    f"len(slow_tokenizer) = {len(slow_tokenizer)}."
                )
                result["warnings"].append(warning_message)
                if not keep_fast_tokenizer:
                    fast_tokenizer = None
                slow_tokenizer = None

        return fast_tokenizer, slow_tokenizer

    tokenizers = []
    feature_extractors = []
    for processor in processors:
        if isinstance(processor, PreTrainedTokenizerBase):
            if processor.__class__.__name__ not in {x.__class__.__name__ for x in tokenizers}:
                tokenizers.append(processor)
        elif isinstance(processor, BaseImageProcessor):
            if processor.__class__.__name__ not in {x.__class__.__name__ for x in feature_extractors}:
                feature_extractors.append(processor)
        elif isinstance(processor, FeatureExtractionMixin):
            if processor.__class__.__name__ not in {x.__class__.__name__ for x in feature_extractors}:
                feature_extractors.append(processor)
        elif isinstance(processor, ProcessorMixin):
            if hasattr(processor, "tokenizer"):
                if processor.tokenizer.__class__.__name__ not in {x.__class__.__name__ for x in tokenizers}:
                    tokenizers.append(processor.tokenizer)
            # Currently, we only have these 2 possibilities
            if hasattr(processor, "image_processor"):
                if processor.image_processor.__class__.__name__ not in {
                    x.__class__.__name__ for x in feature_extractors
                }:
                    feature_extractors.append(processor.image_processor)
            elif hasattr(processor, "feature_extractor"):
                if processor.feature_extractor.__class__.__name__ not in {
                    x.__class__.__name__ for x in feature_extractors
                }:
                    feature_extractors.append(processor.feature_extractor)

    # check the built processors have the unique type
    num_types = len({x.__class__.__name__ for x in feature_extractors})
    if num_types >= 2:
        raise ValueError(f"`feature_extractors` should contain at most 1 type, but it contains {num_types} types!")
    num_types = len({x.__class__.__name__.replace("Fast", "") for x in tokenizers})
    if num_types >= 2:
        raise ValueError(f"`tokenizers` should contain at most 1 tokenizer type, but it contains {num_types} types!")

    fast_tokenizer = None
    slow_tokenizer = None

    for tokenizer in tokenizers:
        if isinstance(tokenizer, PreTrainedTokenizerFast):
            fast_tokenizer = tokenizer
        else:
            slow_tokenizer = tokenizer

    # If the (original) fast/slow tokenizers don't correspond, keep only the fast tokenizer.
    # This doesn't necessarily imply the fast/slow tokenizers in a single Hub repo. has issues.
    # It's more of an issue in `build_processor` which tries to get a checkpoint with as much effort as possible.
    # For `YosoModel` (which uses `AlbertTokenizer(Fast)`), its real (Hub) checkpoint doesn't contain valid files to
    # load the slower tokenizer (`AlbertTokenizer`), and it ends up finding the (canonical) checkpoint of `AlbertModel`,
    # which has different vocabulary.
    # TODO: Try to improve `build_processor`'s definition and/or usage to avoid the above situation in the first place.
    fast_tokenizer, slow_tokenizer = _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=True)
    original_fast_tokenizer, original_slow_tokenizer = fast_tokenizer, slow_tokenizer

    if fast_tokenizer:
        try:
            # Wav2Vec2ForCTC , ByT5Tokenizer etc. all are already small enough and have no fast version that can
            # be retrained
            if fast_tokenizer.vocab_size > TARGET_VOCAB_SIZE:
                fast_tokenizer = convert_tokenizer(fast_tokenizer)
        except Exception:
            result["warnings"].append(
                (
                    f"Failed to convert the fast tokenizer for {fast_tokenizer.__class__.__name__}.",
                    traceback.format_exc(),
                )
            )

    # If `fast_tokenizer` exists, `slow_tokenizer` should correspond to it.
    if fast_tokenizer:
        # Make sure the fast tokenizer can be saved
        try:
            # We don't save it to `output_folder` at this moment - only at the end of this function.
            with tempfile.TemporaryDirectory() as tmpdir:
                fast_tokenizer.save_pretrained(tmpdir)
                try:
                    slow_tokenizer = AutoTokenizer.from_pretrained(tmpdir, use_fast=False)
                except Exception:
                    result["warnings"].append(
                        (
                            f"Failed to load the slow tokenizer saved from {fast_tokenizer.__class__.__name__}.",
                            traceback.format_exc(),
                        )
                    )
                    # Let's just keep the fast version
                    slow_tokenizer = None
        except Exception:
            result["warnings"].append(
                (
                    f"Failed to save the fast tokenizer for {fast_tokenizer.__class__.__name__}.",
                    traceback.format_exc(),
                )
            )
            fast_tokenizer = None

    # If the (possibly converted) fast/slow tokenizers don't correspond, set them to `None`, and use the original
    # tokenizers.
    fast_tokenizer, slow_tokenizer = _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=False)

    # If there is any conversion failed, we keep the original tokenizers.
    if (original_fast_tokenizer is not None and fast_tokenizer is None) or (
        original_slow_tokenizer is not None and slow_tokenizer is None
    ):
        warning_messagae = (
            "There are some issues when converting the fast/slow tokenizers. The original tokenizers from the Hub "
            " will be used instead."
        )
        result["warnings"].append(warning_messagae)
        # Let's use the original version at the end (`original_fast_tokenizer` and `original_slow_tokenizer`)
        fast_tokenizer = original_fast_tokenizer
        slow_tokenizer = original_slow_tokenizer

    # Make sure the fast tokenizer can be saved
    if fast_tokenizer:
        # We don't save it to `output_folder` at this moment - only at the end of this function.
        with tempfile.TemporaryDirectory() as tmpdir:
            try:
                fast_tokenizer.save_pretrained(tmpdir)
            except Exception:
                result["warnings"].append(
                    (
                        f"Failed to save the fast tokenizer for {fast_tokenizer.__class__.__name__}.",
                        traceback.format_exc(),
                    )
                )
                fast_tokenizer = None
    # Make sure the slow tokenizer can be saved
    if slow_tokenizer:
        # We don't save it to `output_folder` at this moment - only at the end of this function.
        with tempfile.TemporaryDirectory() as tmpdir:
            try:
                slow_tokenizer.save_pretrained(tmpdir)
            except Exception:
                result["warnings"].append(
                    (
                        f"Failed to save the slow tokenizer for {slow_tokenizer.__class__.__name__}.",
                        traceback.format_exc(),
                    )
                )
                slow_tokenizer = None

    # update feature extractors using the tiny config
    try:
        feature_extractors = [convert_feature_extractor(p, tiny_config) for p in feature_extractors]
    except Exception:
        result["warnings"].append(
            (
                "Failed to convert feature extractors.",
                traceback.format_exc(),
            )
        )
        feature_extractors = []

    if hasattr(tiny_config, "max_position_embeddings") and tiny_config.max_position_embeddings > 0:
        if fast_tokenizer is not None:
            if fast_tokenizer.__class__.__name__ in [
                "RobertaTokenizerFast",
                "XLMRobertaTokenizerFast",
                "LongformerTokenizerFast",
                "MPNetTokenizerFast",
            ]:
                fast_tokenizer.model_max_length = tiny_config.max_position_embeddings - 2
            else:
                fast_tokenizer.model_max_length = tiny_config.max_position_embeddings
        if slow_tokenizer is not None:
            if slow_tokenizer.__class__.__name__ in [
                "RobertaTokenizer",
                "XLMRobertaTokenizer",
                "LongformerTokenizer",
                "MPNetTokenizer",
            ]:
                slow_tokenizer.model_max_length = tiny_config.max_position_embeddings - 2
            else:
                slow_tokenizer.model_max_length = tiny_config.max_position_embeddings

    processors = [fast_tokenizer, slow_tokenizer] + feature_extractors
    processors = [p for p in processors if p is not None]
    for p in processors:
        p.save_pretrained(output_folder)



SUMMARY:
Here's a concise summary:

The code creates and saves a model with modified tokenizers (fast and slow) when using the Hugging Face library. It performs the following tasks:

1. Checks available tokenizers on the Hugging Face Hub.
2. Converts fast and slow tokenizers to smaller versions based on a provided `tiny_config`.
3. Verifies that the converted tokenizers correspond to each other.
4. Saves modified feature extractors (if applicable) using the tiny config.
5. Calls back the modified processors to ensure they can be saved in their updated form.

The code maintains the functionality of original tokenizers and saves only modified versions, if conversions are successful.


key: chroma:document
string_value:
CODE:

def get_checkpoint_dir(output_dir, model_arch):
    """Get framework-agnostic architecture name. Used to save all PT/TF/Flax models into the same directory."""

    arch_name = model_arch.__name__
    if arch_name.startswith("TF"):
        arch_name = arch_name[2:]
    elif arch_name.startswith("Flax"):
        arch_name = arch_name[4:]



SUMMARY:
**Summary**

The `get_checkpoint_dir` function creates a framework-agnostic architecture name by removing the framework prefix ("TF" or "Flax") from the model architecture. It then joins this modified name with the provided output directory using `os.path.join`. 

**Key Details:**

* Function purpose: Returns a framework-agnostic checkpoint directory.
* Implementation:
	+ Removes framework prefix from model architecture (`arch_name`).
	+ Uses `os.path.join` to combine filename and directory.

**Dependency:**

* Imports an externally-used module: `os`.
* Assumes the existence of a string object containing model architecture information.
* Relies on string manipulation for framework-agnostic namingConvention.


key: chroma:document
string_value:
CODE:

def build_model(model_arch, tiny_config, output_dir):
    """Create and save a model for `model_arch`.

    Also copy the set of processors to each model (under the same model type) output folder.
    """

    checkpoint_dir = get_checkpoint_dir(output_dir, model_arch)

    processor_output_dir = os.path.join(output_dir, "processors")
    # copy the (same set of) processors (for a model type) to the model arch. specific folder
    if os.path.isdir(processor_output_dir):
        shutil.copytree(processor_output_dir, checkpoint_dir, dirs_exist_ok=True)

    tiny_config = copy.deepcopy(tiny_config)

    if any(model_arch.__name__.endswith(x) for x in ["ForCausalLM", "LMHeadModel"]):
        tiny_config.is_encoder_decoder = False
        tiny_config.is_decoder = True

    model = model_arch(config=tiny_config)
    model.save_pretrained(checkpoint_dir)
    model.from_pretrained(checkpoint_dir)



SUMMARY:
Here is a concise and focused code summary:

**Functionality:**

* Creates and saves a model for the specified `model_arch` using a custom configuration (`tiny_config`) and output directory (`output_dir`)
* Copies processor files to the same folder as the checkpoint directory if it exists
* Saves and loads the pre-trained model with the specified `checkpoint_dir`

**Key Implementation Details:**

* Uses `shutil.copytree` to copy processors, allowing for reuse across models of the same type
* Applies settings from `tiny_config` based on the model architecture name (`model_arch.__name__`)

**Notable Dependencies and Requirements:**

* Hugged library (not explicitly listed, but implied by usage)
* os library (for file system operations)
* shutil library (used for copying files)


key: chroma:document
string_value:
CODE:

def fill_result_with_error(result, error, trace, models_to_create):
    """Fill `result` with errors for all target model arch if we can't build processor"""
    error = (error, trace)
    result["error"] = error
    for framework in FRAMEWORKS:
        if framework in models_to_create:
            result[framework] = {}
            for model_arch in models_to_create[framework]:
                result[framework][model_arch.__name__] = {"model": None, "checkpoint": None, "error": error}



SUMMARY:
**Summary:**

`fill_result_with_error` function:
```python
def fill_result_with_error(result, error, trace, models_to_create):
    """
    Fill `result` with errors for various framework models and processor.

    Parameters:
    result (dict): Target dictionary to be filled with errors.
    error (tuple): Error message and traceback.
    trace (traceback object): Debugging information.
    models_to_create (dict): Models to create for each framework.
    """
```

**Functionality:**

1. Fills `result` dictionary with errors for all target model architectures (model_arch in `models_to_create`) if the processor cannot be built.
2. Initializes processor configuration with only class names for each component.

**Key Implementation Details:**

* Iterates through `FRAMEWORKS` and updates corresponding framework models in `result`.
* Uses nested dictionaries to store model configurations for each architecture.

**Notable Dependencies or Requirements:**

* `FRAMEWORKS` (list of strings)
* `models_to_create` (dictionary with framework models)
* `traceback` object (`trace`)
* Processor components with their class names

This summary aims to provide a concise overview of the function's purpose, key implementation details, and any notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

def upload_model(model_dir, organization, token):
    """Upload the tiny models"""

    arch_name = model_dir.split(os.path.sep)[-1]
    repo_name = f"tiny-random-{arch_name}"
    repo_id = f"{organization}/{repo_name}"

    repo_exist = False
    error = None
    try:
        create_repo(repo_id=repo_id, exist_ok=False, repo_type="model", token=token)
    except Exception as e:
        error = e
        if "You already created" in str(e):
            error = None
            logger.warning("Remote repository exists and will be cloned.")
            repo_exist = True
            try:
                create_repo(repo_id=repo_id, exist_ok=True, repo_type="model", token=token)
            except Exception as e:
                error = e
    if error is not None:
        raise error

    with tempfile.TemporaryDirectory() as tmpdir:
        repo = Repository(local_dir=tmpdir, clone_from=repo_id, token=token)
        repo.git_pull()
        shutil.copytree(model_dir, tmpdir, dirs_exist_ok=True)

        if repo_exist:
            # Open a PR on the existing Hub repo.
            hub_pr_url = upload_folder(
                folder_path=model_dir,
                repo_id=repo_id,
                repo_type="model",
                commit_message=f"Update tiny models for {arch_name}",
                commit_description=f"Upload tiny models for {arch_name}",
                create_pr=True,
                token=token,
            )
            logger.warning(f"PR open in {hub_pr_url}.")
            # TODO: We need this information?
        else:
            # Push to Hub repo directly
            repo.git_add(auto_lfs_track=True)
            repo.git_commit(f"Upload tiny models for {arch_name}")
            repo.git_push(blocking=True)  # this prints a progress bar with the upload


SUMMARY:
**Summary:**

This function, `upload_model`, uploads a small model library (`model_dir`) to a remote repository on Hub. The primary purpose is to:

1. Create or clone an existing repository in Hub (`create_repo` method) if it doesn't exist or already exists and needs to be updated.
2. Import the models from the local directory and update the repository with the latest changes.

**Key Implementation Details:**

* Creates a temporary directory and clones the remote repository using `Repository.git_pull()`.
* Copies the model directory into the temporary directory.
* If the repository exists, opens a Pull Request (PR) on Hub (`upload_folder` method).
* If not, directly pushes the updated models to the Hub repository.

**Notable Dependencies or Requirements:**

* `Hub` API (Remote Repository creation/cloning and PR creation)
* `Repository` class for interacting with the remote repository
* `TemporaryDirectory` from `tempfile` module for creating a temporary directory
* `shutil` for copying files and directories
* `os.path.sep` for splitting the model directory path

**Method Signatures and Interfaces:**

* `upload_model(model_dir, organization, token)`
* `create_repo(repo_id, exist_ok=False, repo_type="model", token=token)`
* `Repository(local_dir=tmpdir, clone_from=repo_id, token=token)`
* `upload_folder(folder_path=model_dir, repo_id=repo_id, repo_type="model", ...)`
* `logger.warning` and similar logging functions for output


key: chroma:document
string_value:
CODE:

def build_composite_models(config_class, output_dir):
    import tempfile

    from transformers import (
        BertConfig,
        BertLMHeadModel,
        BertModel,
        BertTokenizer,
        BertTokenizerFast,
        EncoderDecoderModel,
        GPT2Config,
        GPT2LMHeadModel,
        GPT2Tokenizer,
        GPT2TokenizerFast,
        SpeechEncoderDecoderModel,
        TFEncoderDecoderModel,
        TFVisionEncoderDecoderModel,
        TFVisionTextDualEncoderModel,
        VisionEncoderDecoderModel,
        VisionTextDualEncoderModel,
        ViTConfig,
        ViTFeatureExtractor,
        ViTModel,
        Wav2Vec2Config,
        Wav2Vec2Model,
        Wav2Vec2Processor,
    )

    # These will be removed at the end if they are empty
    result = {"error": None, "warnings": []}

    if config_class.model_type == "encoder-decoder":
        encoder_config_class = BertConfig
        decoder_config_class = BertConfig
        encoder_processor = (BertTokenizerFast, BertTokenizer)
        decoder_processor = (BertTokenizerFast, BertTokenizer)
        encoder_class = BertModel
        decoder_class = BertLMHeadModel
        model_class = EncoderDecoderModel
        tf_model_class = TFEncoderDecoderModel
    elif config_class.model_type == "vision-encoder-decoder":
        encoder_config_class = ViTConfig
        decoder_config_class = GPT2Config
        encoder_processor = (ViTFeatureExtractor,)
        decoder_processor = (GPT2TokenizerFast, GPT2Tokenizer)
        encoder_class = ViTModel
        decoder_class = GPT2LMHeadModel
        model_class = VisionEncoderDecoderModel
        tf_model_class = TFVisionEncoderDecoderModel
    elif config_class.model_type == "speech-encoder-decoder":
        encoder_config_class = Wav2Vec2Config
        decoder_config_class = BertConfig
        encoder_processor = (Wav2Vec2Processor,)
        decoder_processor = (BertTokenizerFast, BertTokenizer)
        encoder_class = Wav2Vec2Model
        decoder_class = BertLMHeadModel
        model_class = SpeechEncoderDecoderModel
        tf_model_class = None
    elif config_class.model_type == "vision-text-dual-encoder":
        # Not encoder-decoder, but encoder-encoder. We just keep the same name as above to make code easier
        encoder_config_class = ViTConfig
        decoder_config_class = BertConfig
        encoder_processor = (ViTFeatureExtractor,)
        decoder_processor = (BertTokenizerFast, BertTokenizer)
        encoder_class = ViTModel
        decoder_class = BertModel
        model_class = VisionTextDualEncoderModel
        tf_model_class = TFVisionTextDualEncoderModel

    with tempfile.TemporaryDirectory() as tmpdir:
        try:
            # build encoder
            models_to_create = {"processor": encoder_processor, "pytorch": (encoder_class,), "tensorflow": []}
            encoder_output_dir = os.path.join(tmpdir, "encoder")
            build(encoder_config_class, models_to_create, encoder_output_dir)

            # build decoder
            models_to_create = {"processor": decoder_processor, "pytorch": (decoder_class,), "tensorflow": []}
            decoder_output_dir = os.path.join(tmpdir, "decoder")
            build(decoder_config_class, models_to_create, decoder_output_dir)

            # build encoder-decoder
            encoder_path = os.path.join(encoder_output_dir, encoder_class.__name__)
            decoder_path = os.path.join(decoder_output_dir, decoder_class.__name__)

            if config_class.model_type != "vision-text-dual-encoder":
                # Specify these explicitly for encoder-decoder like models, but not for `vision-text-dual-encoder` as it
                # has no decoder.
                decoder_config = decoder_config_class.from_pretrained(decoder_path)
                decoder_config.is_decoder = True
                decoder_config.add_cross_attention = True
                model = model_class.from_encoder_decoder_pretrained(
                    encoder_path,
                    decoder_path,
                    decoder_config=decoder_config,
                )
            elif config_class.model_type == "vision-text-dual-encoder":
                model = model_class.from_vision_text_pretrained(encoder_path, decoder_path)

            model_path = os.path.join(
                output_dir,
                f"{model_class.__name__}-{encoder_config_class.model_type}-{decoder_config_class.model_type}",
            )
            model.save_pretrained(model_path)

            if tf_model_class is not None:
                model = tf_model_class.from_pretrained(model_path)
                model.save_pretrained(model_path)

            # copy the processors
            encoder_processor_path = os.path.join(encoder_output_dir, "processors")
            decoder_processor_path = os.path.join(decoder_output_dir, "processors")
            if os.path.isdir(encoder_processor_path):
                shutil.copytree(encoder_processor_path, model_path, dirs_exist_ok=True)
            if os.path.isdir(decoder_processor_path):
                shutil.copytree(decoder_processor_path, model_path, dirs_exist_ok=True)

            # fill `result`
            result["processor"] = {x.__name__: x.__name__ for x in encoder_processor + decoder_processor}

            result["pytorch"] = {model_class.__name__: {"model": model_class.__name__, "checkpoint": model_path}}

            result["tensorflow"] = {}
            if tf_model_class is not None:
                result["tensorflow"] = {
                    tf_model_class.__name__: {"model": tf_model_class.__name__, "checkpoint": model_path}
                }
        except Exception:
            result["error"] = (
                f"Failed to build models for {config_class.__name__}.",
                traceback.format_exc(),
            )

    if not result["error"]:
        del result["error"]
    if not result["warnings"]:
        del result["warnings"]



SUMMARY:
**Summary**

The `build_composite_models` function builds composite models from various pre-trained transformer architectures. Its primary purpose is to create a full-fledged encoder-decoder or vision-text-dual-encoder model from its input configuration, pre-trained models, and output directory.

**Key Implementation Details:**

*   The function relies on the Hugging Face Transformers library.
*   It handles three types of composite models:
    *   Encoder-decoder (`config_class.model_type == "encoder-decoder"`): Uses PyTorch.
    *   Vision-encoder-decoder (`config_class.model_type == "vision-encoder-decoder"`): Uses PyTorch.
    *   Speech-encoder-decoder (`config_class.model_type == "speech-encoder-decoder"`): Supports both PyTorch and TensorFlow.
    *   Vision-text-dual-encoder (`config_class.model_type == "vision-text-dual-encoder"`): Only uses PyTorch.
*   Pre-trained models are loaded, configured, and combined to form the final composite model using the `model_from_pretrained`, `model_from_encoder_decoder`, or `model_from_vision_text` methods.

**Notable Dependencies:**

*   The `transformers` library from Hugging Face.
*   Temporary directories for storing pre-built models.
*   External functions and classes (e.g., `build`, `os.path.join`, `shutil.copytree`) from Python's standard library or other dependencies not explicitly mentioned here.

**Model Output:**

The function generates a directory with a composite model, along with various output files, including:
        *   Model checkpoint
        *   PyTorch models and checkpoints (when supported)
        *   TensorFlow models and checkpoints (if specified)
        *   PyTorch processors

**Interfaces:**

*   `build`: An external function for building pre-trained models.
*   `TransformerModel` interface from the Hugging Face Transformers library.

```python
def build_composite_models(config_class, output_dir):
    # ... [function implementation]
    
    return result
```

This concise summary should provide essential information for developers to understand this code snippet and replicate its functionality in their own projects.


key: chroma:document
string_value:
CODE:

def get_token_id_from_tokenizer(token_id_name, tokenizer, original_token_id):
    """Use `tokenizer` to get the values of `bos_token_id`, `eos_token_ids`, etc.

    The argument `token_id_name` should be a string ending with `_token_id`, and `original_token_id` should be an
    integer that will be return if `tokenizer` has no token corresponding to `token_id_name`.
    """

    token_id = original_token_id

    if not token_id_name.endswith("_token_id"):
        raise ValueError(f"`token_id_name` is {token_id_name}, which doesn't end with `_token_id`!")

    token = getattr(tokenizer, token_id_name.replace("_token_id", "_token"), None)
    if token is not None:
        if isinstance(tokenizer, PreTrainedTokenizerFast):
            token_id = tokenizer._convert_token_to_id_with_added_voc(token)
        else:
            token_id = tokenizer._convert_token_to_id(token)



SUMMARY:
**Summary**

`get_token_id_from_tokenizer` function retrieves the ID of a specific token from a `tokenizer` object. It:

* Takes in `token_id_name` (string) and `original_token_id` (integer)
* Initializes `token_id` to `original_token_id`
* Raises ValueError if `token_id_name` does not end with `_token_id`
* Uses `tokenizer` to retrieve the token ID, optionally converting it with added vocabulary

**Key Implementation Details**

* Replaces `_token` in `token_id_name` with `_token` from `tokenizer` for attribute access
* Checks if `tokenizer` is a `PreTrainedTokenizerFast` and applies conversion using its `_convert_token_to_id_with_added_voc` method

**Notable Dependencies or Requirements**

* External dependencies:
	+ `transformers` library (specifically, `PreTrainedTokenizerFast`)
* Related classes/functions:
	+ ` tokenizer._convert_token_to_id`
	+ `tokenizer._convert_token_to_id_with_added_voc`


key: chroma:document
string_value:
CODE:

def get_config_overrides(config_class, processors):
    # `Bark` configuration is too special. Let's just not handle this for now.
    if config_class.__name__ == "BarkConfig":
        return {}

    config_overrides = {}

    # Check if there is any tokenizer (prefer fast version if any)
    tokenizer = None
    for processor in processors:
        if isinstance(processor, PreTrainedTokenizerFast):
            tokenizer = processor
            break
        elif isinstance(processor, PreTrainedTokenizer):
            tokenizer = processor

    if tokenizer is None:
        return config_overrides

    # Get some properties of the (already converted) tokenizer (smaller vocab size, special token ids, etc.)
    # We use `len(tokenizer)` instead of `tokenizer.vocab_size` to avoid potential issues for tokenizers with non-empty
    # `added_tokens_encoder`. One example is the `DebertaV2Tokenizer` where the mask token is the extra token.
    vocab_size = len(tokenizer)

    # The original checkpoint has length `35998`, but it doesn't have ids `30400` and `30514` but instead `35998` and
    # `35999`.
    if config_class.__name__ == "GPTSanJapaneseConfig":
        vocab_size += 2

    config_overrides["vocab_size"] = vocab_size

    # Used to create a new model tester with `tokenizer.vocab_size` in order to get the (updated) special token ids.
    model_tester_kwargs = {"vocab_size": vocab_size}
    # `FSMTModelTester` accepts `src_vocab_size` and `tgt_vocab_size` but not `vocab_size`.
    if config_class.__name__ == "FSMTConfig":
        del model_tester_kwargs["vocab_size"]
        model_tester_kwargs["src_vocab_size"] = tokenizer.src_vocab_size
        model_tester_kwargs["tgt_vocab_size"] = tokenizer.tgt_vocab_size

    _tiny_config = get_tiny_config(config_class, **model_tester_kwargs)

    # handle the possibility of `text_config` inside `_tiny_config` for clip-like models (`owlvit`, `groupvit`, etc.)
    if hasattr(_tiny_config, "text_config"):
        _tiny_config = _tiny_config.text_config

    # Collect values of some special token ids
    for attr in dir(_tiny_config):
        if attr.endswith("_token_id"):
            token_id = getattr(_tiny_config, attr)
            if token_id is not None:
                # Using the token id values from `tokenizer` instead of from `_tiny_config`.
                token_id = get_token_id_from_tokenizer(attr, tokenizer, original_token_id=token_id)
                config_overrides[attr] = token_id

    if config_class.__name__ == "FSMTConfig":
        config_overrides["src_vocab_size"] = tokenizer.src_vocab_size
        config_overrides["tgt_vocab_size"] = tokenizer.tgt_vocab_size
        # `FSMTConfig` has `DecoderConfig` as `decoder` attribute.
        config_overrides["decoder"] = configuration_fsmt.DecoderConfig(
            vocab_size=tokenizer.tgt_vocab_size, bos_token_id=config_overrides["eos_token_id"]
        )



SUMMARY:
**Function Summary: `get_configOverrides`**

**Purpose:** Returns configuration overrides for a given `config_class` and list of `processors`, specifically designed for Hugging Face models.

**Key Implementation Details:**

* Checks if a specific tokenizer is present in the `processors` list and returns overrides accordingly.
* Extracts properties from the chosen tokenizer, including vocabulary size, special token ids, and decoder configuration.
* Generates model tester keyword arguments based on the chosen tokenizer's vocabulary size and decoder attributes.
* Iterates over special token id attributes and collects their values using the tokenizer or `_tiny_config` object.

**Notable Dependencies and Requirements:**

* `PreTrainedTokenizerFast` and `PreTrainedTokenizer`
* External dependencies:
	+ `FSMTModelTester`
	+ `configuration_fsmt`
	+ `DebertaV2Tokenizer`
* Inherited/Implemented interfaces:
	+ `config_class`: uses specific classes from the Hugging Face library.
	+ `configuration_fsmt`: used to create a decoder configuration for FSMTConfig models.


key: chroma:document
string_value:
CODE:

def build(config_class, models_to_create, output_dir):
    """Create all models for a certain model type.

    Args:
        config_class (`PretrainedConfig`):
            A subclass of `PretrainedConfig` that is used to determine `models_to_create`.
        models_to_create (`dict`):
            A dictionary containing the processor/model classes that we want to create the instances. These models are
            of the same model type which is associated to `config_class`.
        output_dir (`str`):
            The directory to save all the checkpoints. Each model architecture will be saved in a subdirectory under
            it. Models in different frameworks with the same architecture will be saved in the same subdirectory.
    """
    if data["training_ds"] is None or data["testing_ds"] is None:
        ds = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1")
        data["training_ds"] = ds["train"]
        data["testing_ds"] = ds["test"]

    if config_class.model_type in [
        "encoder-decoder",
        "vision-encoder-decoder",
        "speech-encoder-decoder",
        "vision-text-dual-encoder",
    ]:
        return build_composite_models(config_class, output_dir)

    result = {k: {} for k in models_to_create}

    # These will be removed at the end if they are empty
    result["error"] = None
    result["warnings"] = []

    # Build processors
    processor_classes = models_to_create["processor"]

    if len(processor_classes) == 0:
        error = f"No processor class could be found in {config_class.__name__}."
        fill_result_with_error(result, error, None, models_to_create)
        logger.error(result["error"][0])
        return result

    for processor_class in processor_classes:
        try:
            processor = build_processor(config_class, processor_class, allow_no_checkpoint=True)
            if processor is not None:
                result["processor"][processor_class] = processor
        except Exception:
            error = f"Failed to build processor for {processor_class.__name__}."
            trace = traceback.format_exc()
            fill_result_with_error(result, error, trace, models_to_create)
            logger.error(result["error"][0])
            return result

    if len(result["processor"]) == 0:
        error = f"No processor could be built for {config_class.__name__}."
        fill_result_with_error(result, error, None, models_to_create)
        logger.error(result["error"][0])
        return result

    try:
        tiny_config = get_tiny_config(config_class)
    except Exception as e:
        error = f"Failed to get tiny config for {config_class.__name__}: {e}"
        trace = traceback.format_exc()
        fill_result_with_error(result, error, trace, models_to_create)
        logger.error(result["error"][0])
        return result

    # Convert the processors (reduce vocabulary size, smaller image size, etc.)
    processors = list(result["processor"].values())
    processor_output_folder = os.path.join(output_dir, "processors")
    try:
        processors = convert_processors(processors, tiny_config, processor_output_folder, result)
    except Exception:
        error = "Failed to convert the processors."
        trace = traceback.format_exc()
        result["warnings"].append((error, trace))

    if len(processors) == 0:
        error = f"No processor is returned by `convert_processors` for {config_class.__name__}."
        fill_result_with_error(result, error, None, models_to_create)
        logger.error(result["error"][0])
        return result

    try:
        config_overrides = get_config_overrides(config_class, processors)
    except Exception as e:
        error = f"Failure occurs while calling `get_config_overrides`: {e}"
        trace = traceback.format_exc()
        fill_result_with_error(result, error, trace, models_to_create)
        logger.error(result["error"][0])
        return result

    # Just for us to see this easily in the report
    if "vocab_size" in config_overrides:
        result["vocab_size"] = config_overrides["vocab_size"]

    # Update attributes that `vocab_size` involves
    for k, v in config_overrides.items():
        if hasattr(tiny_config, k):
            setattr(tiny_config, k, v)
        # So far, we only have to deal with `text_config`, as `config_overrides` contains text-related attributes only.
        # `FuyuConfig` saves data under both FuyuConfig and its `text_config`. This is not good, but let's just update
        # every involved fields to avoid potential failure.
        if (
            hasattr(tiny_config, "text_config")
            and tiny_config.text_config is not None
            and hasattr(tiny_config.text_config, k)
        ):
            setattr(tiny_config.text_config, k, v)
            # If `text_config_dict` exists, we need to update its value here too in order to # make
            # `save_pretrained -> from_pretrained` work.
            if hasattr(tiny_config, "text_config_dict"):
                tiny_config.text_config_dict[k] = v

    if result["warnings"]:
        logger.warning(result["warnings"][0][0])

    # update `result["processor"]`
    result["processor"] = {type(p).__name__: p.__class__.__name__ for p in processors}

    for pytorch_arch in models_to_create["pytorch"]:
        result["pytorch"][pytorch_arch.__name__] = {}
        error = None
        try:
            model = build_model(pytorch_arch, tiny_config, output_dir=output_dir)
        except Exception as e:
            model = None
            error = f"Failed to create the pytorch model for {pytorch_arch}: {e}"
            trace = traceback.format_exc()

        result["pytorch"][pytorch_arch.__name__]["model"] = model.__class__.__name__ if model is not None else None
        result["pytorch"][pytorch_arch.__name__]["checkpoint"] = (
            get_checkpoint_dir(output_dir, pytorch_arch) if model is not None else None
        )
        if error is not None:
            result["pytorch"][pytorch_arch.__name__]["error"] = (error, trace)
            logger.error(f"{pytorch_arch.__name__}: {error}")

    for tensorflow_arch in models_to_create["tensorflow"]:
        # Make PT/TF weights compatible
        pt_arch_name = tensorflow_arch.__name__[2:]  # Remove `TF`
        pt_arch = getattr(transformers_module, pt_arch_name)

        result["tensorflow"][tensorflow_arch.__name__] = {}
        error = None
        if pt_arch.__name__ in result["pytorch"] and result["pytorch"][pt_arch.__name__]["checkpoint"] is not None:
            ckpt = get_checkpoint_dir(output_dir, pt_arch)
            # Use the same weights from PyTorch.
            try:
                model = tensorflow_arch.from_pretrained(ckpt)
                model.save_pretrained(ckpt)
            except Exception as e:
                # Conversion may fail. Let's not create a model with different weights to avoid confusion (for now).
                model = None
                error = f"Failed to convert the pytorch model to the tensorflow model for {pt_arch}: {e}"
                trace = traceback.format_exc()
        else:
            try:
                model = build_model(tensorflow_arch, tiny_config, output_dir=output_dir)
            except Exception as e:
                model = None
                error = f"Failed to create the tensorflow model for {tensorflow_arch}: {e}"
                trace = traceback.format_exc()

        result["tensorflow"][tensorflow_arch.__name__]["model"] = (
            model.__class__.__name__ if model is not None else None
        )
        result["tensorflow"][tensorflow_arch.__name__]["checkpoint"] = (
            get_checkpoint_dir(output_dir, tensorflow_arch) if model is not None else None
        )
        if error is not None:
            result["tensorflow"][tensorflow_arch.__name__]["error"] = (error, trace)
            logger.error(f"{tensorflow_arch.__name__}: {error}")

    if not result["error"]:
        del result["error"]
    if not result["warnings"]:
        del result["warnings"]



SUMMARY:
**Summary**

This is the `build` function, which creates PyTorch and TensorFlow models for a given `config_class`, `models_to_create`, and `output_dir`. The primary purpose is to build and save checkpoints for multiple model types.

**Main Functionality**

1. Creates processors using `build_processor` with the given `processor_classes`.
2. Converts processors by reducing vocabulary size and image size.
3. Get tiny config using `get_tiny_config`.
4. Build and save PyTorch models using `build_model` for each PyTorch arch.
5. Build and save TensorFlow models using `from_pretrained` and `save_pretrained` methods for each TensorFlow arch.

**Output**

The function returns a dictionary containing the following:
- `processor`: Model types and their corresponding names
- `pytorch`: PyTorch models and their checkpoints
- `tensorflow`: TensorFlow models and their checkpoints

**Key Functions**

1. `build_processor`
2. `convert_processors`
3. `get_tiny_config`
4. `build_model` (PyTorch)
5. `from_pretrained` (TensorFlow)

Note: The function contains many nested sections and conditional statements, making it complex and hard to read. Some sections are commented out or not relevant to the current task, which may make it difficult to understand the entire codebase.


key: chroma:document
string_value:
CODE:

def build_tiny_model_summary(results, organization=None, token=None):
    """Build a summary: a dictionary of the form
    {
      model architecture name:
        {
          "tokenizer_classes": [...],
          "processor_classes": [...],
          "model_classes": [...],
        }
      ..
    }
    """
    tiny_model_summary = {}
    for config_name in results:
        processors = [key for key, value in results[config_name]["processor"].items()]
        tokenizer_classes = sorted([x for x in processors if x.endswith("TokenizerFast") or x.endswith("Tokenizer")])
        processor_classes = sorted([x for x in processors if x not in tokenizer_classes])
        for framework in FRAMEWORKS:
            if framework not in results[config_name]:
                continue
            for arch_name in results[config_name][framework]:
                model_classes = [arch_name]
                base_arch_name = arch_name[2:] if arch_name.startswith("TF") else arch_name
                # tiny model is not created for `arch_name`
                if results[config_name][framework][arch_name]["model"] is None:
                    model_classes = []
                if base_arch_name not in tiny_model_summary:
                    tiny_model_summary[base_arch_name] = {}
                tiny_model_summary[base_arch_name].update(
                    {
                        "tokenizer_classes": tokenizer_classes,
                        "processor_classes": processor_classes,
                    }
                )
                tiny_model_summary[base_arch_name]["model_classes"] = sorted(
                    tiny_model_summary[base_arch_name].get("model_classes", []) + model_classes
                )
                if organization is not None:
                    repo_name = f"tiny-random-{base_arch_name}"
                    # composite models' checkpoints have more precise repo. names on the Hub.
                    if base_arch_name in COMPOSITE_MODELS:
                        repo_name = f"tiny-random-{COMPOSITE_MODELS[base_arch_name]}"
                    repo_id = f"{organization}/{repo_name}"
                    try:
                        commit_hash = hf_api.repo_info(repo_id, token=token).sha
                    except Exception:
                        # The directory is not created, but processor(s) is/are included in `results`.
                        logger.warning(f"Failed to get information for {repo_id}.\n{traceback.format_exc()}")
                        del tiny_model_summary[base_arch_name]
                        continue
                    tiny_model_summary[base_arch_name]["sha"] = commit_hash



SUMMARY:
**Function Summary:**

`build_tiny_model_summary`

Builds a dictionary of model architecture summaries.

**Purpose and Functionality:**
The function takes in `results`, `organization`, and `token`, generating a summary dictionary with model names as keys and sub-dictionaries containing tokenizer classes, processor classes, model classes, and commit SHA hashes as values. It iterates through configurations in `results` and extracts models from different frameworks (TF/HT/TF2), updating the model summary accordingly.

**Key Implementation Details:**

* Iteration over configurations in `results`
* Extractions of tokenizer classes (`x.endswith("TokenizerFast") or x.endswith("Tokenizer")`)
* Extractions of processor classes (`not in tokenizer_classes`)
* Iteration over frameworks (TF/HT/TF2) and architecture names
* Model class updates for each architecture name

**Notable Dependencies and Requirements:**

* `hf_api` library for accessing repository information
* Various external configurations stored in `results`
* Optional `organization` parameter with repository ID generation
* Token (`token`) used for authentication


key: chroma:document
string_value:
CODE:

def build_failed_report(results, include_warning=True):
    failed_results = {}
    for config_name in results:
        if "error" in results[config_name]:
            if config_name not in failed_results:
                failed_results[config_name] = {}
            failed_results[config_name] = {"error": results[config_name]["error"]}

        if include_warning and "warnings" in results[config_name]:
            if config_name not in failed_results:
                failed_results[config_name] = {}
            failed_results[config_name]["warnings"] = results[config_name]["warnings"]

        for framework in FRAMEWORKS:
            if framework not in results[config_name]:
                continue
            for arch_name in results[config_name][framework]:
                if "error" in results[config_name][framework][arch_name]:
                    if config_name not in failed_results:
                        failed_results[config_name] = {}
                    if framework not in failed_results[config_name]:
                        failed_results[config_name][framework] = {}
                    if arch_name not in failed_results[config_name][framework]:
                        failed_results[config_name][framework][arch_name] = {}
                    error = results[config_name][framework][arch_name]["error"]
                    failed_results[config_name][framework][arch_name]["error"] = error



SUMMARY:
**Function Summary:**
`build_failed_report(results, include_warning=True)` generates a dictionary containing detailed error reports for each configuration in the `results` input.

**Purpose and Functionality:**

* Iterates through configurations with error reports, framework-specific arch-names, and warnings (if requested).
* Identifies and extracts applicable error messages from "error" or "warnings" sub-dictionary.
* Creates a nested dictionary structure with failed results, including configuration name, framework, architecture name, and corresponding error message.

**Key Implementation Details:**

* Iterates through `FRAMEWORKS` list to process arch-names under each framework.
* Uses conditional statements to navigate the complex data structure of "error" and/or "warnings" sub-dictionary.

**Notable Dependencies or Requirements:**

* `FRAMEWORKS` namespace (list of supported frameworks)
* Access to dictionary keys `"error"` and/or `"warnings"` within specific configurations.

**Method Signatures and Interfaces:**

* takes two inputs:
	+ `results`: input dictionary containing configuration-specific data.
	+ `include_warning`: optional boolean flag (default: True).
	+ returns: a nested dictionary `failed_results` containing error reports for each configuration.


key: chroma:document
string_value:
CODE:

def build_simple_report(results):
    text = ""
    failed_text = ""
    for config_name in results:
        for framework in FRAMEWORKS:
            if framework not in results[config_name]:
                continue
            for arch_name in results[config_name][framework]:
                if "error" in results[config_name][framework][arch_name]:
                    result = results[config_name][framework][arch_name]["error"]
                    failed_text += f"{arch_name}: {result[0]}\n"
                else:
                    result = ("OK",)
                text += f"{arch_name}: {result[0]}\n"



SUMMARY:
**Summary**

The `build_simple_report` function generates two report strings: `text` and `failed_text`. It iterates through a nested dictionary of test results to generate a simple text report. The primary purpose is to extract and format pass/fail information for each framework, architecture, and configuration.

**Key Implementation Details**

* Iterates through three loops to gather data:
	+ Outer loop: Configuration names (`config_name`)
	+ Middle loop: Frameworks (`framework` from `FRAMEWORKS`)
	+ Inner loop: Architecture names (`arch_name`) within a given framework
* Checks for "error" in the result, and formats either an OK or failed message
* Formats and concatenates results into two separate strings using `text` and `failed_text`

**Notable Dependencies/Requirements**

* `FRAMEWORKS`: An iterable of list-like objects representing frameworks to report on
* Related classes/functions not shown in this snippet


key: chroma:document
string_value:
CODE:

def update_tiny_model_summary_file(report_path):
    with open(os.path.join(report_path, "tiny_model_summary.json")) as fp:
        new_data = json.load(fp)
    with open("tests/utils/tiny_model_summary.json") as fp:
        data = json.load(fp)
    for key, value in new_data.items():
        if key not in data:
            data[key] = value
        else:
            for attr in ["tokenizer_classes", "processor_classes", "model_classes"]:
                # we might get duplication here. We will remove them below when creating `updated_data`.
                data[key][attr].extend(value[attr])
            new_sha = value.get("sha", None)
            if new_sha is not None:
                data[key]["sha"] = new_sha

    updated_data = {}
    for key in sorted(data.keys()):
        updated_data[key] = {}
        for attr, value in data[key].items():
            # deduplication and sort
            updated_data[key][attr] = sorted(set(value)) if attr != "sha" else value

    with open(os.path.join(report_path, "updated_tiny_model_summary.json"), "w") as fp:


SUMMARY:
**Concise Summary:**

The `update_tiny_model_summary_file` function updates the contents of two JSON files, comparing data from two sources and replacing duplicates or updating specific attributes. The main purpose is to merge and validate tiny model summaries.

* **Primary Purpose:** Merge data from two sources into a unified summary.
* **Key Implementation Details:**
	+ Iterates over duplicate key-value pairs, updating the lower priority value with the higher priority one.
	+ Appends values from lists while preserving duplicates, except for the SHA hash attribute.
	+ Sorts and deduplicates list values, excluding the SHA hash field.
* **Notable Dependencies/Requirements:**
	+ `json` module for parsing file contents
	+ `os` module for path manipulation
	+ Access to files "tiny_model_summary.json" and "updated_tiny_model_summary.json".

**RAG System Notes:**


key: chroma:document
string_value:
CODE:

def create_tiny_models(
    output_path,
    all,
    model_types,
    models_to_skip,
    no_check,
    upload,
    organization,
    token,
    num_workers=1,
):
    clone_path = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
    if os.getcwd() != clone_path:
        raise ValueError(f"This script should be run from the root of the clone of `transformers` {clone_path}")

    report_path = os.path.join(output_path, "reports")
    os.makedirs(report_path, exist_ok=True)

    _pytorch_arch_mappings = [
        x
        for x in dir(transformers_module)
        if x.startswith("MODEL_") and x.endswith("_MAPPING") and x != "MODEL_NAMES_MAPPING"
    ]
    _tensorflow_arch_mappings = [
        x for x in dir(transformers_module) if x.startswith("TF_MODEL_") and x.endswith("_MAPPING")
    ]

    pytorch_arch_mappings = [getattr(transformers_module, x) for x in _pytorch_arch_mappings]
    tensorflow_arch_mappings = [getattr(transformers_module, x) for x in _tensorflow_arch_mappings]

    config_classes = CONFIG_MAPPING.values()
    if not all:
        config_classes = [CONFIG_MAPPING[model_type] for model_type in model_types]

    # A map from config classes to tuples of processors (tokenizer, feature extractor, processor) classes
    processor_type_map = {c: get_processor_types_from_config_class(c) for c in config_classes}

    to_create = {}
    for c in config_classes:
        processors = processor_type_map[c]
        models = get_architectures_from_config_class(c, pytorch_arch_mappings, models_to_skip)
        tf_models = get_architectures_from_config_class(c, tensorflow_arch_mappings, models_to_skip)
        if len(models) + len(tf_models) > 0:
            to_create[c] = {"processor": processors, "pytorch": models, "tensorflow": tf_models}

    results = {}
    if num_workers <= 1:
        for c, models_to_create in list(to_create.items()):
            print(f"Create models for {c.__name__} ...")
            result = build(c, models_to_create, output_dir=os.path.join(output_path, c.model_type))
            results[c.__name__] = result
            print("=" * 40)
    else:
        all_build_args = []
        for c, models_to_create in list(to_create.items()):
            all_build_args.append((c, models_to_create, os.path.join(output_path, c.model_type)))
        with multiprocessing.Pool() as pool:
            results = pool.starmap(build, all_build_args)
            results = {buid_args[0].__name__: result for buid_args, result in zip(all_build_args, results)}

    if upload:
        if organization is None:
            raise ValueError("The argument `organization` could not be `None`. No model is uploaded")

        to_upload = []
        for model_type in os.listdir(output_path):
            # This is the directory containing the reports
            if model_type == "reports":
                continue
            for arch in os.listdir(os.path.join(output_path, model_type)):
                if arch == "processors":
                    continue
                to_upload.append(os.path.join(output_path, model_type, arch))
        to_upload = sorted(to_upload)

        upload_results = {}
        if len(to_upload) > 0:
            for model_dir in to_upload:
                try:
                    upload_model(model_dir, organization, token)
                except Exception as e:
                    error = f"Failed to upload {model_dir}. {e.__class__.__name__}: {e}"
                    logger.error(error)
                    upload_results[model_dir] = error

        with open(os.path.join(report_path, "failed_uploads.json"), "w") as fp:
            json.dump(upload_results, fp, indent=4)

    # Build the tiny model summary file. The `tokenizer_classes` and `processor_classes` could be both empty lists.
    # When using the items in this file to update the file `tests/utils/tiny_model_summary.json`, the model
    # architectures with `tokenizer_classes` and `processor_classes` being both empty should **NOT** be added to
    # `tests/utils/tiny_model_summary.json`.
    tiny_model_summary = build_tiny_model_summary(results, organization=organization, token=token)
    with open(os.path.join(report_path, "tiny_model_summary.json"), "w") as fp:
        json.dump(tiny_model_summary, fp, indent=4)

    with open(os.path.join(report_path, "tiny_model_creation_report.json"), "w") as fp:
        json.dump(results, fp, indent=4)

    # Build the warning/failure report (json format): same format as the complete `results` except this contains only
    # warnings or errors.
    failed_results = build_failed_report(results)
    with open(os.path.join(report_path, "failed_report.json"), "w") as fp:
        json.dump(failed_results, fp, indent=4)

    simple_report, failed_report = build_simple_report(results)
    # The simplified report: a .txt file with each line of format:
    # {model architecture name}: {OK or error message}
    with open(os.path.join(report_path, "simple_report.txt"), "w") as fp:
        fp.write(simple_report)

    # The simplified failure report: same above except this only contains line with errors
    with open(os.path.join(report_path, "simple_failed_report.txt"), "w") as fp:
        fp.write(failed_report)



SUMMARY:
**Function Summary**

The `create_tiny_models` function creates and builds models for various transformer architectures. Its primary purpose is to generate reports on the creation process.

**Key Functionality:**

1.  Checks if the current directory is the root of the cloned 'transformers' repository.
2.  Creates a "reports" directory in the output path and sets it up with proper permissions.
3.  Lists all the model types, processes it through `model_types` list or retrieves them from all by using CONFIG_MAPPING.values() if required, and then maps to specific architectures like 'tokenizer_classes' and 'processor_classes'.
4.  If num_workers is greater than 1, uses multiprocessing pool with star map function to build models in parallel.
5.  Builds the tiny model summary file based on results.

**Implementation Details:**

-   Uses `os.path` module to work with paths.
-   Utilizes the dir() function of objects to get specific architectures like pytorch and tensorflow models from transformers_module.

**Notable Dependencies or Requirements:**

1.  The functions builds on top of the os, multiprocessing and json modules as shown below:
    -   `os`: used for working with file paths.
    -   `multiprocessing.Pool` : used to build models in parallel.
    -   `json`: used to dump results in JSON format.

2.  References transformers_module which is not a custom module but likely part of an existing library or framework, containing classes and methods like "build" as mentioned above which builds models and also processes to_create dictionary which maps config class names to processors and architectures built based on pytorch arch mappings.

3.  The organization and token are requirements used during upload process.
4.  It uses a custom function build_tiny_model_summary where model type, tokenizer_classes, processor_classes are passed as arguments.


key: chroma:document
string_value:
CODE:
# Function to perform topological sorting
def topological_sort(dependencies: dict):
    # Nodes are the name of the models to convert (we only add those to the graph)
    nodes = {node.rsplit("modular_", 1)[1].replace(".py", "") for node in dependencies.keys()}
    # This will be a graph from models to convert, to models to convert that should be converted before (as they are a dependency)
    graph = {}
    name_mapping = {}
    for node, deps in dependencies.items():
        node_name = node.rsplit("modular_", 1)[1].replace(".py", "")
        dep_names = {dep.split(".")[-2] for dep in deps}
        dependencies = {dep for dep in dep_names if dep in nodes and dep != node_name}
        graph[node_name] = dependencies
        name_mapping[node_name] = node

    sorting_list = []
    while len(graph) > 0:
        # Find the nodes with 0 out-degree
        leaf_nodes = {node for node in graph if len(graph[node]) == 0}
        # Add them to the list
        sorting_list += list(leaf_nodes)
        # Remove the leafs from the graph (and from the deps of other nodes)
        graph = {node: deps - leaf_nodes for node, deps in graph.items() if node not in leaf_nodes}



SUMMARY:
**Concise Summary**

The `topological_sort` function performs a topological sort on a graph of model dependencies. It takes a dictionary `dependencies` as input, where each key is a node (model) and its value is a list of its dependent models.

**Key Features:**

* The function constructs a directed graph from the dependencies.
* It uses a depth-first search algorithm to find nodes with no outgoing edges (leaf nodes), which are then added to a sorting list.
* The process repeats until all nodes have been visited, resulting in a sorted list of model names corresponding to the original input dictionary keys.

**Notes:**

* External dependencies include `graph` and `name_mapping`, which represent temporary data structures used during the sorting process.
* No external libraries are required beyond Python's built-in `dict` and set operations.


key: chroma:document
string_value:
CODE:
# Function to extract class and import info from a file
def extract_classes_and_imports(file_path):
    with open(file_path, "r") as file:
        tree = ast.parse(file.read(), filename=file_path)
    imports = set()

    for node in ast.walk(tree):
        if isinstance(node, (ast.Import, ast.ImportFrom)):
            module = node.module if isinstance(node, ast.ImportFrom) else None
            if module and (".modeling_" in module or "transformers.models" in module):
                imports.add(module)


SUMMARY:
**Concise Summary**

The `extract_classes_and_imports` function extracts classes and imports from a Python file. It:

* Parses the file into an Abstract Syntax Tree (AST)
*walks through the AST to identify import nodes containing ".modeling_" or "transformers.models" modules
*Returns a set of unique imported module paths.

**Key Implementation Details**

The key implementation details are:

* The use of `ast.Import` and `ast.ImportFrom` node types to identify imports.
* The use of `filename` parameter in `ast.parse()` to preserve the file path.
* The `walk()` function to traverse the AST tree.

**Notable Dependencies or Requirements**

The function relies on the following dependencies:

* Python's built-in `ast` module (Abstract Syntax Tree).
* No external libraries are required.


key: chroma:document
string_value:
CODE:
# Function to map dependencies between classes
def map_dependencies(py_files):
    dependencies = defaultdict(set)
    # First pass: Extract all classes and map to files
    for file_path in py_files:
        # dependencies[file_path].add(None)
        class_to_file = extract_classes_and_imports(file_path)
        for module in class_to_file:
            dependencies[file_path].add(module)


SUMMARY:
**Summary**

 Maps dependencies between Python file locations. 

**Key Implementation Details**

* Merges `extract_classes_and_imports` function call into a single loop, iterating over each file in `py_files`
* Utilizes a `defaultdict(set)` to efficiently store and manage dependencies for each file
* Returns a dictionary mapping file paths to sets of imported modules

**Notable Dependencies or Requirements**

* None explicitly mentioned; no external dependencies are referenced. 

(Note: The exact implementation of the `extract_classes_and_imports` function is not shown in this snippet, but it presumably extracts class definitions and import statements from a given Python file.)


key: chroma:document
string_value:
CODE:

def find_priority_list(py_files):
    dependencies = map_dependencies(py_files)
    ordered_classes = topological_sort(dependencies)


SUMMARY:
**Summary:**

This function, `find_priority_list`, takes a list of Python files (`py_files`) as input. It maps dependencies between classes, performs a topological sort on the dependency graph, and returns an ordered list of classes.

**Key Details:**

* Maps class dependencies using `map_dependencies(py_files)`
* Performs topological sort on dependency graph using `topological_sort(dependencies)`
* Returns ordered list of classes

**Notable Dependencies/Requirements:**

* None explicitly stated, but `map_dependencies` and `topological_sort` likely rely on external libraries (e.g., Pyrex, graphlib) that are not included in this function summary.


key: chroma:document
string_value:
CODE:

def find_indent(line: str) -> int:
    """
    Returns the number of spaces that start a line indent.
    """
    search = re.search(r"^(\s*)(?:\S|$)", line)
    if search is None:
        return 0


SUMMARY:
**Summary:**

`find_indent(line: str) -> int`: Returns the number of spaces at the start of a line, used for indentation calculation.

**Key Details:**

* Uses `re.search` to search for a pattern of one or more non-breaking spaces (`\s+`) at the beginning of the line (`^`).
* If no match is found, returns 0.
* Returns the length (i.e., number of spaces) in the matched group.

**Example Usage:**


key: chroma:document
string_value:
CODE:

def stringify_default(default: Any) -> str:
    """
    Returns the string representation of a default value, as used in docstring: numbers are left as is, all other
    objects are in backtiks.

    Args:
        default (`Any`): The default value to process

    Returns:
        `str`: The string representation of that default.
    """
    if isinstance(default, bool):
        # We need to test for bool first as a bool passes isinstance(xxx, (int, float))
        return f"`{default}`"
    elif isinstance(default, enum.Enum):
        # We need to test for enum first as an enum with int values will pass isinstance(xxx, (int, float))
        return f"`{str(default)}`"
    elif isinstance(default, int):
        return str(default)
    elif isinstance(default, float):
        result = str(default)
        return str(round(default, 2)) if len(result) > 6 else result
    elif isinstance(default, str):
        return str(default) if default.isnumeric() else f'`"{default}"`'
    elif isinstance(default, type):
        return f"`{default.__name__}`"
    else:


SUMMARY:
**Summary:**

This function `stringify_default` converts its input `default` value into a string representation. It handles various data types:

*   Boolean (`True/False`) and integers as is
*   Enums using their representation as a string
*   Floats with precise 2-decimal places formatting (unless more than 6 characters long)
*   Strings with numeric values or enclosed in backticks

**Key implementation details:**

| Implementation | Description |
| --- | --- |
| `bool` and `int` conversions | Directly return as strings |
| `enum.Enum` conversion | Use `str()` to convert to string, preserving enum value |
| Float formatting | Use `round()` with 2 decimal places for strings longer than 6 characters |
| String handling | Separate numeric strings from non-numeric strings and apply formatting accordingly |

**Noteable dependencies:**

*   `enum`: Used to handle Enums
*   `str()`, `round()`: Implemented in Python's built-in functions

This summary provides a concise overview of the function's primary functionality, key implementation details, and notable dependencies.


key: chroma:document
string_value:
CODE:

def eval_math_expression(expression: str) -> Optional[Union[float, int]]:
    # Mainly taken from the excellent https://stackoverflow.com/a/9558001
    """
    Evaluate (safely) a mathematial expression and returns its value.

    Args:
        expression (`str`): The expression to evaluate.

    Returns:
        `Optional[Union[float, int]]`: Returns `None` if the evaluation fails in any way and the value computed
        otherwise.

    Example:

    ```py
    >>> eval_expr('2^6')
    4
    >>> eval_expr('2**6')
    64
    >>> eval_expr('1 + 2*3**(4^5) / (6 + -7)')
    -5.0
    ```
    """
    try:
        return eval_node(ast.parse(expression, mode="eval").body)
    except TypeError:


SUMMARY:
**Summary**

This function, `eval_math_expression`, evaluates a mathematical expression provided as a string and returns its value. It safely handles various mathematical operations, including exponentiation, multiplication, division, and addition/subtraction.

**Key Implementation Details**

* Utilizes the `eval` function from Python's standard library to evaluate the expression.
* Parses the expression using `ast.parse` with the `mode="eval"` argument.
* Calls the `eval_node` function (not shown in this code snippet) on the parsed AST.

**Notable Dependencies and Requirements**

* Python's standard library, specifically the `ast` module for parsing.
* The `eval_node` function is not part of the standard library and is assumed to be implemented separately.


key: chroma:document
string_value:
CODE:

def eval_node(node):
    if isinstance(node, ast.Num):  # <number>
        return node.n
    elif isinstance(node, ast.BinOp):  # <left> <operator> <right>
        return MATH_OPERATORS[type(node.op)](eval_node(node.left), eval_node(node.right))
    elif isinstance(node, ast.UnaryOp):  # <operator> <operand> e.g., -1
        return MATH_OPERATORS[type(node.op)](eval_node(node.operand))
    else:


SUMMARY:
**Eval Node Function Summary**

### Purpose and Functionality

The `eval_node` function is a recursive traverser of abstract syntax tree (AST) nodes, evaluating arithmetic expressions by applying mathematical operators to their operands.

### Key Implementation Details

* Evaluates integer literal nodes (`ast.Num`) as-is.
* Unifies binary operation nodes (`ast.BinOp`): recursively evaluates left and right operands with operator-dependent functions from `MATH_OPERATORS`.
* Unifies unary operation nodes (`ast.UnaryOp`): recursively evaluates the operand with an operator-dependent function from `MATH_OPERATORS`.
* Raises a `TypeError` for unsupported node types.

### Dependencies

None, as this implementation does not rely on external dependencies. However, it utilizes:

* `ast` (Abstract Syntax Tree) module for parsing expressions.
* `MATH Operators` dictionary (not shown in provided code), which maps operator types to evaluation functions.
* No external libraries besides the Python standard library.


key: chroma:document
string_value:
CODE:

def replace_default_in_arg_description(description: str, default: Any) -> str:
    """
    Catches the default value in the description of an argument inside a docstring and replaces it by the value passed.

    Args:
        description (`str`): The description of an argument in a docstring to process.
        default (`Any`): The default value that whould be in the docstring of that argument.

    Returns:
       `str`: The description updated with the new default value.
    """
    # Lots of docstrings have `optional` or **opational** instead of *optional* so we do this fix here.
    description = description.replace("`optional`", OPTIONAL_KEYWORD)
    description = description.replace("**optional**", OPTIONAL_KEYWORD)
    if default is inspect._empty:
        # No default, make sure the description doesn't have any either
        idx = description.find(OPTIONAL_KEYWORD)
        if idx != -1:
            description = description[:idx].rstrip()
            if description.endswith(","):
                description = description[:-1].rstrip()
    elif default is None:
        # Default None are not written, we just set `*optional*`. If there is default that is not None specified in the
        # description, we do not erase it (as sometimes we set the default to `None` because the default is a mutable
        # object).
        idx = description.find(OPTIONAL_KEYWORD)
        if idx == -1:
            description = f"{description}, {OPTIONAL_KEYWORD}"
        elif re.search(r"defaults to `?None`?", description) is not None:
            len_optional = len(OPTIONAL_KEYWORD)
            description = description[: idx + len_optional]
    else:
        str_default = None
        # For numbers we may have a default that is given by a math operation (1/255 is really popular). We don't
        # want to replace those by their actual values.
        if isinstance(default, (int, float)) and re.search("defaults to `?(.*?)(?:`|$)", description) is not None:
            # Grab the default and evaluate it.
            current_default = re.search("defaults to `?(.*?)(?:`|$)", description).groups()[0]
            if default == eval_math_expression(current_default):
                try:
                    # If it can be directly converted to the type of the default, it's a simple value
                    str_default = str(type(default)(current_default))
                except Exception:
                    # Otherwise there is a math operator so we add a code block.
                    str_default = f"`{current_default}`"
            elif isinstance(default, enum.Enum) and default.name == current_default.split(".")[-1]:
                # When the default is an Enum (this is often the case for PIL.Image.Resampling), and the docstring
                # matches the enum name, keep the existing docstring rather than clobbering it with the enum value.
                str_default = f"`{current_default}`"

        if str_default is None:
            str_default = stringify_default(default)
        # Make sure default match
        if OPTIONAL_KEYWORD not in description:
            description = f"{description}, {OPTIONAL_KEYWORD}, defaults to {str_default}"
        elif _re_parse_description.search(description) is None:
            idx = description.find(OPTIONAL_KEYWORD)
            len_optional = len(OPTIONAL_KEYWORD)
            description = f"{description[:idx + len_optional]}, defaults to {str_default}"
        else:
            description = _re_parse_description.sub(rf"*optional*, defaults to {str_default}", description)



SUMMARY:
**Summary:**

Function `replace_default_in_arg_description` updates the argument description in a docstring to include the provided default value. It performs various fixes and transformations based on the type and content of the default value.

**Key Features:**

* Replaces occurrences of "optional" or "**optional**" with `OPTIONAL_KEYWORD`.
* Handles cases where the default value is `None`, `empty string` (e.g., empty line), or a complex math expression.
* For numbers, evaluates the math expression if it matches the default value and converts it to the correct type.

**Important Variables and Functions:**

* `OPTIONAL_KEYWORD`: "optional" or "**optional**"
* `description`: input docstring
* `default`: default value to be included in the updated description

**Notable Dependencies:**

* `inspect` for checking `default` value
* `re` for regular expression matching and replacing
* `eval_math_expression` function not provided in the code snippet, but used for evaluating math expressions

**Return Value:**

The updated docstring with the default value included.


key: chroma:document
string_value:
CODE:

def get_default_description(arg: inspect.Parameter) -> str:
    """
    Builds a default description for a parameter that was not documented.

    Args:
        arg (`inspect.Parameter`): The argument in the signature to generate a description for.

    Returns:
        `str`: The description.
    """
    if arg.annotation is inspect._empty:
        arg_type = "<fill_type>"
    elif hasattr(arg.annotation, "__name__"):
        arg_type = arg.annotation.__name__
    else:
        arg_type = str(arg.annotation)

    if arg.default is inspect._empty:
        return f"`{arg_type}`"
    elif arg.default is None:
        return f"`{arg_type}`, {OPTIONAL_KEYWORD}"
    else:
        str_default = stringify_default(arg.default)


SUMMARY:
**get_default_description function summary**

```markdown
Provides default description for an uncovered argument in Python function signatures.

### Key implementation details:

* Uses `inspect.Parameter` and related types (`inspect._empty`) to determine parameter type and defaults.
* Employs string formatting to construct the final default description.

### Notable dependencies and interfaces:

* `inspect`: Imported module providing type inspection utilities.
* `optional`: Likely an external library (e.g., `pydantic`) for optional parameter handling, referenced by `OPTIONAL_KEYWORD`.
```

### Primary purpose and functionality
Builds a default description string for an uncovered argument in a Python function signature, considering its type and potential default value.


key: chroma:document
string_value:
CODE:

def find_source_file(obj: Any) -> Path:
    """
    Finds the source file of an object.

    Args:
        obj (`Any`): The object whose source file we are looking for.

    Returns:
        `Path`: The source file.
    """
    module = obj.__module__
    obj_file = PATH_TO_TRANSFORMERS
    for part in module.split(".")[1:]:
        obj_file = obj_file / part


SUMMARY:
**Summary:**

Finds the source file of an object given its `__module__` attribute.

*   **Purpose:** Returns the path to the source file associated with the input object's module.
*   **Implementation Details:**
    *   Uses the `__module__` attribute of the input object to determine the module name.
    *   Joins the module name with a prefix (`PATH_TO_TRANSFORMERS`) to form the initial directory path.
    *   Iterates through the parts of the module name (skipping the first part, which is the package name), appending each part as a directory.
        *   `module.split(".")[1:]`: Extracts the namespace from the module name after removing the package prefix ('__main__.').
    *   Returns the final path with a `.py` suffix using `Path.with_suffix()`.
*   **Dependencies:** None explicitly mentioned; however, it relies on the `PATH_TO_TRANSFORMERS` constant and possibly `Path`.

**Key Variables/Functions:**

*   `obj`: Input object with its `__module__` attribute.
*   `module`: The namespace of the input object's module name (e.g., 'transformers.modeling.tokenization').
*   `PARTS OF THE MODULE NAME`: An array consisting of module parts excluding the package index prefix (`__main__.`).


key: chroma:document
string_value:
CODE:

def match_docstring_with_signature(obj: Any) -> Optional[Tuple[str, str]]:
    """
    Matches the docstring of an object with its signature.

    Args:
        obj (`Any`): The object to process.

    Returns:
        `Optional[Tuple[str, str]]`: Returns `None` if there is no docstring or no parameters documented in the
        docstring, otherwise returns a tuple of two strings: the current documentation of the arguments in the
        docstring and the one matched with the signature.
    """
    if len(getattr(obj, "__doc__", "")) == 0:
        # Nothing to do, there is no docstring.
        return

    # Read the docstring in the source code to see if there is a special command to ignore this object.
    try:
        source, _ = inspect.getsourcelines(obj)
    except OSError:
        source = []

    idx = 0
    while idx < len(source) and '"""' not in source[idx]:
        idx += 1

    ignore_order = False
    if idx < len(source):
        line_before_docstring = source[idx - 1]
        if re.search(r"^\s*#\s*no-format\s*$", line_before_docstring):
            # This object is ignored
            return
        elif re.search(r"^\s*#\s*ignore-order\s*$", line_before_docstring):
            ignore_order = True

    # Read the signature
    signature = inspect.signature(obj).parameters

    obj_doc_lines = obj.__doc__.split("\n")
    # Get to the line where we start documenting arguments
    idx = 0
    while idx < len(obj_doc_lines) and _re_args.search(obj_doc_lines[idx]) is None:
        idx += 1

    if idx == len(obj_doc_lines):
        # Nothing to do, no parameters are documented.
        return

    indent = find_indent(obj_doc_lines[idx])
    arguments = {}
    current_arg = None
    idx += 1
    start_idx = idx
    # Keep going until the arg section is finished (nonempty line at the same indent level) or the end of the docstring.
    while idx < len(obj_doc_lines) and (
        len(obj_doc_lines[idx].strip()) == 0 or find_indent(obj_doc_lines[idx]) > indent
    ):
        if find_indent(obj_doc_lines[idx]) == indent + 4:
            # New argument -> let's generate the proper doc for it
            re_search_arg = _re_parse_arg.search(obj_doc_lines[idx])
            if re_search_arg is not None:
                _, name, description = re_search_arg.groups()
                current_arg = name
                if name in signature:
                    default = signature[name].default
                    if signature[name].kind is inspect._ParameterKind.VAR_KEYWORD:
                        default = None
                    new_description = replace_default_in_arg_description(description, default)
                else:
                    new_description = description
                init_doc = _re_parse_arg.sub(rf"\1\2 ({new_description}):", obj_doc_lines[idx])
                arguments[current_arg] = [init_doc]
        elif current_arg is not None:
            arguments[current_arg].append(obj_doc_lines[idx])

        idx += 1

    # We went too far by one (perhaps more if there are a lot of new lines)
    idx -= 1
    if current_arg:
        while len(obj_doc_lines[idx].strip()) == 0:
            arguments[current_arg] = arguments[current_arg][:-1]
            idx -= 1
    # And we went too far by one again.
    idx += 1

    old_doc_arg = "\n".join(obj_doc_lines[start_idx:idx])

    old_arguments = list(arguments.keys())
    arguments = {name: "\n".join(doc) for name, doc in arguments.items()}
    # Add missing arguments with a template
    for name in set(signature.keys()) - set(arguments.keys()):
        arg = signature[name]
        # We ignore private arguments or *args/**kwargs (unless they are documented by the user)
        if name.startswith("_") or arg.kind in [
            inspect._ParameterKind.VAR_KEYWORD,
            inspect._ParameterKind.VAR_POSITIONAL,
        ]:
            arguments[name] = ""
        else:
            arg_desc = get_default_description(arg)
            arguments[name] = " " * (indent + 4) + f"{name} ({arg_desc}): <fill_docstring>"

    # Arguments are sorted by the order in the signature unless a special comment is put.
    if ignore_order:
        new_param_docs = [arguments[name] for name in old_arguments if name in signature]
        missing = set(signature.keys()) - set(old_arguments)
        new_param_docs.extend([arguments[name] for name in missing if len(arguments[name]) > 0])
    else:
        new_param_docs = [arguments[name] for name in signature.keys() if len(arguments[name]) > 0]
    new_doc_arg = "\n".join(new_param_docs)



SUMMARY:
**Function Summary:**
### match_docstring_with_signature(obj)
 
  **Purpose:** Match the docstring of an object with its signature.

  **Primary Functionality:** The function `match_docstring_with_signature` compares two documentation strings for an object's methods and arguments. It reads the source code (if available) to check for ignore commands, then parses the signatures using `inspect.signature`. It extracts argument names and descriptions from the docstring, generating new documentations based on the found signature parameters.

  **Key Implementation Details:**
- Uses regular expressions (`re`) for pattern matching.
- Leverages `find_indent` (function not shown in this snippet) to parse indentation levels.
- Utilizes `_pre_parse_arg` and `_parse_arg` regular expression patterns (not directly visible here).

  **Dependencies/Requirements:**
- Import of `inspect`, `re`, and access to `type Any`, `Optional`, `Tuple`, and `inspect._ParameterKind`.
 - The function utilizes type hints for clarity (`Any`, `Inspect.Signature`).
  
  **API Interface:** 
    It accepts an object, returns a tuple containing two strings:
     - The first string contains the original documentation of the arguments described in the docstring.
     - The second string contains the matched and generated documentation for the same argument parameters based on their signature.

**Additional Notes:**
The function seems heavily involved with parsing and manipulating various parts of Python source code, likely to maintain consistency across different codebases by following certain established coding standards.


key: chroma:document
string_value:
CODE:

def fix_docstring(obj: Any, old_doc_args: str, new_doc_args: str):
    """
    Fixes the docstring of an object by replacing its arguments documentaiton by the one matched with the signature.

    Args:
        obj (`Any`):
            The object whose dostring we are fixing.
        old_doc_args (`str`):
            The current documentation of the parameters of `obj` in the docstring (as returned by
            `match_docstring_with_signature`).
        new_doc_args (`str`):
            The documentation of the parameters of `obj` matched with its signature (as returned by
            `match_docstring_with_signature`).
    """
    # Read the docstring in the source code and make sure we have the right part of the docstring
    source, line_number = inspect.getsourcelines(obj)

    # Get to the line where we start documenting arguments
    idx = 0
    while idx < len(source) and _re_args.search(source[idx]) is None:
        idx += 1

    if idx == len(source):
        # Args are not defined in the docstring of this object
        return

    # Get to the line where we stop documenting arguments
    indent = find_indent(source[idx])
    idx += 1
    start_idx = idx
    while idx < len(source) and (len(source[idx].strip()) == 0 or find_indent(source[idx]) > indent):
        idx += 1

    idx -= 1
    while len(source[idx].strip()) == 0:
        idx -= 1
    idx += 1

    if "".join(source[start_idx:idx])[:-1] != old_doc_args:
        # Args are not fully defined in the docstring of this object
        return

    obj_file = find_source_file(obj)
    with open(obj_file, "r", encoding="utf-8") as f:
        content = f.read()

    # Replace content
    lines = content.split("\n")
    lines = lines[: line_number + start_idx - 1] + [new_doc_args] + lines[line_number + idx - 1 :]

    print(f"Fixing the docstring of {obj.__name__} in {obj_file}.")
    with open(obj_file, "w", encoding="utf-8") as f:


SUMMARY:
**Function Summary: `fix_docstring` Function**

### Purpose and Functionality

The `fix_docstring` function updates the docstrings of objects by replacing outdated parameter documentation with the latest one based on the object's signature. It ensures accuracy and consistency in documentations.

### Key Implementation Details

1.  **Inspecting Docstrings**:

    *   Reads source code from a given object.
    *   Identifies the lines where arguments are defined and replaced.
2.  **Matching Docstring with Signature**:

    *   Uses `match_docstring_with_signature` to determine the updated documentation based on the signature.

### Notable Dependencies

1  **External Dependency**:

    *   `re` for regular expression matching
2  **Inherited/Implemented Interfaces** (None mentioned in this function)

The code snippet fixes the docstrings of objects by replacing outdated parameter documentation with accurate ones. It uses regular expressions to identify lines where arguments are defined, reads source code from an object, and updates its content accordingly.


key: chroma:document
string_value:
CODE:

def check_docstrings(overwrite: bool = False, check_all: bool = False):
    """
    Check docstrings of all public objects that are callables and are documented. By default, only checks the diff.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether to fix inconsistencies or not.
        check_all (`bool`, *optional*, defaults to `False`):
            Whether to check all files.
    """
    module_diff_files = None
    if not check_all:
        module_diff_files = set()
        repo = Repo(PATH_TO_REPO)
        # Diff from index to unstaged files
        for modified_file_diff in repo.index.diff(None):
            if modified_file_diff.a_path.startswith("src/transformers"):
                module_diff_files.add(modified_file_diff.a_path)
        # Diff from index to `main`
        for modified_file_diff in repo.index.diff(repo.refs.main.commit):
            if modified_file_diff.a_path.startswith("src/transformers"):
                module_diff_files.add(modified_file_diff.a_path)
        # quick escape route: if there are no module files in the diff, skip this check
        if len(module_diff_files) == 0:
            return
        print("    Checking docstrings in the following files:" + "\n    - " + "\n    - ".join(module_diff_files))

    failures = []
    hard_failures = []
    to_clean = []
    for name in dir(transformers):
        # Skip objects that are private or not documented.
        if name.startswith("_") or ignore_undocumented(name) or name in OBJECTS_TO_IGNORE:
            continue

        obj = getattr(transformers, name)
        if not callable(obj) or not isinstance(obj, type) or getattr(obj, "__doc__", None) is None:
            continue

        # If we are checking against the diff, we skip objects that are not part of the diff.
        if module_diff_files is not None:
            object_file = find_source_file(getattr(transformers, name))
            object_file_relative_path = "src/" + str(object_file).split("/src/")[1]
            if object_file_relative_path not in module_diff_files:
                continue

        # Check docstring
        try:
            result = match_docstring_with_signature(obj)
            if result is not None:
                old_doc, new_doc = result
            else:
                old_doc, new_doc = None, None
        except Exception as e:
            print(e)
            hard_failures.append(name)
            continue
        if old_doc != new_doc:
            if overwrite:
                fix_docstring(obj, old_doc, new_doc)
            else:
                failures.append(name)
        elif not overwrite and new_doc is not None and ("<fill_type>" in new_doc or "<fill_docstring>" in new_doc):
            to_clean.append(name)

    # Deal with errors
    error_message = ""
    if len(hard_failures) > 0:
        error_message += (
            "The argument part of the docstrings of the following objects could not be processed, check they are "
            "properly formatted."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in hard_failures])
    if len(failures) > 0:
        error_message += (
            "The following objects docstrings do not match their signature. Run `make fix-copies` to fix this. "
            "In some cases, this error may be raised incorrectly by the docstring checker. If you think this is the "
            "case, you can manually check the docstrings and then add the object name to `OBJECTS_TO_IGNORE` in "
            "`utils/check_docstrings.py`."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in failures])
    if len(to_clean) > 0:
        error_message += (
            "The following objects docstrings contain templates you need to fix: search for `<fill_type>` or "
            "`<fill_docstring>`."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in to_clean])

    if len(error_message) > 0:
        error_message = "There was at least one problem when checking docstrings of public objects.\n" + error_message


SUMMARY:
**Summary**

The `check_docstrings` function checks docstrings of public, callable objects in the `transformers` module and its dependencies. It:

1. **Analyze Documentation**: Checks if existing docs match defined signatures.

2. **Fix Potential Issues**: Fixes inconsistencies or warnings reported by the tool.

3. **Report Potential Bugs**: Prints error messages about unmatched signatures, incomplete methods, and potential bugs.

**Key Functionality**

*   `check_all`: Optional flag to check all files.
*   `overwrite`: Optional flag to fix inconsistencies instead of logging failures.

**Implementation Details**

*   Verifies functions match expected signature patterns.
*   Skips private objects or those without docstrings.
*   Raises `ValueError` with error messages when reports detection issues.


key: chroma:document
string_value:
CODE:

def download_and_extract(task, data_dir):
    print(f"Downloading and extracting {task}...")
    data_file = f"{task}.zip"
    urllib.request.urlretrieve(TASK2PATH[task], data_file)
    with zipfile.ZipFile(data_file) as zip_ref:
        zip_ref.extractall(data_dir)
    os.remove(data_file)


SUMMARY:
**Summary:**

`download_and_extract` function downloads and extracts a zip file for a given task, extracting the contents to a specified data directory.

**Implementation Details:**

* Downloads from `taskId`
* Extracts contents using `zipfile.ZipFile.extractall()`
* Deletes temp zip file after extraction (`os.remove(data_file)`)

**Notable Dependencies:**

* `urllib.request.urlretrieve()` for downloading
* `zipfile` module for extracting zip file
* `os` module for deleting temporary files

**Primary Class/Function Names:**

* `task`
* `data_dir`
* `TASK2PATH[task]`: mapping of tasks to download paths


key: chroma:document
string_value:
CODE:

def format_mrpc(data_dir, path_to_data):
    print("Processing MRPC...")
    mrpc_dir = os.path.join(data_dir, "MRPC")
    if not os.path.isdir(mrpc_dir):
        os.mkdir(mrpc_dir)
    if path_to_data:
        mrpc_train_file = os.path.join(path_to_data, "msr_paraphrase_train.txt")
        mrpc_test_file = os.path.join(path_to_data, "msr_paraphrase_test.txt")
    else:
        print("Local MRPC data not specified, downloading data from %s" % MRPC_TRAIN)
        mrpc_train_file = os.path.join(mrpc_dir, "msr_paraphrase_train.txt")
        mrpc_test_file = os.path.join(mrpc_dir, "msr_paraphrase_test.txt")
        urllib.request.urlretrieve(MRPC_TRAIN, mrpc_train_file)
        urllib.request.urlretrieve(MRPC_TEST, mrpc_test_file)
    if not os.path.isfile(mrpc_train_file):
        raise ValueError(f"Train data not found at {mrpc_train_file}")
    if not os.path.isfile(mrpc_test_file):
        raise ValueError(f"Test data not found at {mrpc_test_file}")
    urllib.request.urlretrieve(TASK2PATH["MRPC"], os.path.join(mrpc_dir, "dev_ids.tsv"))

    dev_ids = []
    with open(os.path.join(mrpc_dir, "dev_ids.tsv"), encoding="utf8") as ids_fh:
        for row in ids_fh:
            dev_ids.append(row.strip().split("\t"))

    with open(mrpc_train_file, encoding="utf8") as data_fh, open(
        os.path.join(mrpc_dir, "train.tsv"), "w", encoding="utf8"
    ) as train_fh, open(os.path.join(mrpc_dir, "dev.tsv"), "w", encoding="utf8") as dev_fh:
        header = data_fh.readline()
        train_fh.write(header)
        dev_fh.write(header)
        for row in data_fh:
            label, id1, id2, s1, s2 = row.strip().split("\t")
            if [id1, id2] in dev_ids:
                dev_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1, s2))
            else:
                train_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1, s2))

    with open(mrpc_test_file, encoding="utf8") as data_fh, open(
        os.path.join(mrpc_dir, "test.tsv"), "w", encoding="utf8"
    ) as test_fh:
        header = data_fh.readline()
        test_fh.write("index\t#1 ID\t#2 ID\t#1 String\t#2 String\n")
        for idx, row in enumerate(data_fh):
            label, id1, id2, s1, s2 = row.strip().split("\t")
            test_fh.write("%d\t%s\t%s\t%s\t%s\n" % (idx, id1, id2, s1, s2))


SUMMARY:
**Function Summary: `format_mrpc`**

**Main Purpose and Functionality**
The `format_mrpc` function processes the MRPC (Microsoft Research Paraphrase Corpus) dataset by:

* Creating a directory for the dataset if it does not exist
* Downloading dataset files from a specified URL or using local data if specified
* Validating the presence of required files
* Preprocessing data into training, development, and testing sets

**Key Implementation Details**

* Uses `os.path.join` to construct file paths
* Utilizes `urllib.request.urlretrieve` to download external datasets
* Reads and writes data from/to files using `open` function with specified encoding
* Splits input file rows into individual labels, IDs, and strings

**Notable Dependencies and Requirements**

* Uses `os` module for file system operations
* Utilizes `urllib.request` module for downloading external datasets
* References classes and functions by exact name:
	+ `MRPC_TRAIN`
	+ `TASK2PATH["MRPC"]`

**External Dependencies:**

* `urllib.request`
* `os`
* Python's built-in file I/O operations


key: chroma:document
string_value:
CODE:

def download_diagnostic(data_dir):
    print("Downloading and extracting diagnostic...")
    if not os.path.isdir(os.path.join(data_dir, "diagnostic")):
        os.mkdir(os.path.join(data_dir, "diagnostic"))
    data_file = os.path.join(data_dir, "diagnostic", "diagnostic.tsv")
    urllib.request.urlretrieve(TASK2PATH["diagnostic"], data_file)
    print("\tCompleted!")


SUMMARY:
**Summary:**

The `download_diagnostic` function downloads and extracts diagnostic data from a specified directory. It:

* Creates the 'diagnostic' subdirectory if it does not exist
* Downloads the diagnostic file using `urlretrieve` from the `TASK2PATH["diagnostic"]` location
* Prints confirmation messages after completion

**Key details:**

* Uses `os.path.join` to construct paths for directory and file creation/export
* Implements `urllib.request.urlretrieve` for file download
* Returns no value (indicating a side-effecting function)

**Notable dependencies/requirements:**

* `os.path`
* `urllib.request`
* `TASK2PATH` (dictionary containing path to diagnostic data)


key: chroma:document
string_value:
CODE:

def get_tasks(task_names):
    task_names = task_names.split(",")
    if "all" in task_names:
        tasks = TASKS
    else:
        tasks = []
        for task_name in task_names:
            if task_name not in TASKS:
                raise ValueError(f"Task {task_name} not found!")
            tasks.append(task_name)


SUMMARY:
**Function Summary: `get_tasks`**

The `get_tasks` function retrieves the specified task names from a predefined list of tasks (`TASKS`). If "all" is among the input task names, it returns all available tasks. Otherwise, it checks each task name against the `TASKS` list and appends valid task names to a result list, raising an error for invalid task names.

**Key Implementation Details:**

* Uses string splitting to parse input task names.
* Checks if "all" is present in the input task names.

**Notable Dependencies or Requirements:**

* The `TASKS` list of predefined task names (assuming it's defined elsewhere).
* None explicitly listed, but implicit dependencies on Python's built-in data structures and error handling mechanisms.


key: chroma:document
string_value:
CODE:

def main(arguments):
    parser = argparse.ArgumentParser()
    parser.add_argument("--data_dir", help="directory to save data to", type=str, default="glue_data")
    parser.add_argument(
        "--tasks", help="tasks to download data for as a comma separated string", type=str, default="all"
    )
    parser.add_argument(
        "--path_to_mrpc",
        help="path to directory containing extracted MRPC data, msr_paraphrase_train.txt and msr_paraphrase_text.txt",
        type=str,
        default="",
    )
    args = parser.parse_args(arguments)

    if not os.path.isdir(args.data_dir):
        os.mkdir(args.data_dir)
    tasks = get_tasks(args.tasks)

    for task in tasks:
        if task == "MRPC":
            format_mrpc(args.data_dir, args.path_to_mrpc)
        elif task == "diagnostic":
            download_diagnostic(args.data_dir)
        else:


SUMMARY:
Here is a concise summary of the provided code chunk:

**Main Functionality:**
The `main` function orchestrates data preparation for multiple natural language processing (NLP) tasks. It:

1. Parses command-line arguments for task specifics (`--data_dir`, `--tasks`, and `--path_to_mrpc`) to download and process data.
2. Creates a directory to store the downloaded data if it does not exist.
3. Determines the tasks to be processed based on the comma-separated string provided in `--tasks`.
4. For each task, performs specific actions:
   - MRPC task: Formats MRPC data using `format_mrpc`.
   - Diagnostic task: Downloads diagnostic data using `download_diagnostic`.
   - Other tasks: Downloads and extracts data using `download_and_extract`.

**Key Implementation Details:**

* Utilizes `argparse` for command-line argument parsing.
* Uses the `get_tasks` function to determine the specific tasks to be processed.
* Implements a conditional statement to perform task-specific actions based on the parsed arguments.

**Notable Dependencies or Requirements:**

* `os` module for directory manipulation (e.g., creating directories).
* `argparse` library for command-line argument parsing.
* External dependencies:
	+ `get_tasks` function (not shown in this snippet).
	+ `download_diagnostic`, `format_mrpc`, and `download_and_extract` functions (not shown in this snippet).


key: chroma:document
string_value:
CODE:

def camel_case_split(identifier: str) -> List[str]:
    """
    Split a camel-cased name into words.

    Args:
        identifier (`str`): The camel-cased name to parse.

    Returns:
        `List[str]`: The list of words in the identifier (as seprated by capital letters).

    Example:

    ```py
    >>> camel_case_split("CamelCasedClass")
    ["Camel", "Cased", "Class"]
    ```
    """
    # Regex thanks to https://stackoverflow.com/questions/29916065/how-to-do-camelcase-split-in-python
    matches = re.finditer(".+?(?:(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|$)", identifier)


SUMMARY:
** camel_case_split() Function Summary **


*   **Purpose and Functionality:** 
    Splits a camel-cased string into individual words based on capitalization.

        *   Type Hint: `identifier (str) -> List[str]`
        *   Returns: A list of words separated by capital letters.

*   **Implementation Details:** 
    - Uses regular expressions with look-behind and look-ahead assertions to match word boundaries.
    - Utilizes the `re.finditer` function to extract matches from the input string.


*   **Notable Dependencies or Requirements:**
    - `re` module for regex functionality
    - No external dependencies


        Example usage:

            ```py
>>> camel_case_split("CamelCasedClass")
['Camel', 'Cased', 'Class']
```


key: chroma:document
string_value:
CODE:

def get_frameworks_table() -> pd.DataFrame:
    """
    Generates a dataframe containing the supported auto classes for each model type, using the content of the auto
    modules.
    """
    # Dictionary model names to config.
    config_maping_names = transformers_module.models.auto.configuration_auto.CONFIG_MAPPING_NAMES
    model_prefix_to_model_type = {
        config.replace("Config", ""): model_type for model_type, config in config_maping_names.items()
    }

    # Dictionaries flagging if each model prefix has a backend in PT/TF/Flax.
    pt_models = collections.defaultdict(bool)
    tf_models = collections.defaultdict(bool)
    flax_models = collections.defaultdict(bool)

    # Let's lookup through all transformers object (once) and find if models are supported by a given backend.
    for attr_name in dir(transformers_module):
        lookup_dict = None
        if _re_tf_models.match(attr_name) is not None:
            lookup_dict = tf_models
            attr_name = _re_tf_models.match(attr_name).groups()[0]
        elif _re_flax_models.match(attr_name) is not None:
            lookup_dict = flax_models
            attr_name = _re_flax_models.match(attr_name).groups()[0]
        elif _re_pt_models.match(attr_name) is not None:
            lookup_dict = pt_models
            attr_name = _re_pt_models.match(attr_name).groups()[0]

        if lookup_dict is not None:
            while len(attr_name) > 0:
                if attr_name in model_prefix_to_model_type:
                    lookup_dict[model_prefix_to_model_type[attr_name]] = True
                    break
                # Try again after removing the last word in the name
                attr_name = "".join(camel_case_split(attr_name)[:-1])

    all_models = set(list(pt_models.keys()) + list(tf_models.keys()) + list(flax_models.keys()))
    all_models = list(all_models)
    all_models.sort()

    data = {"model_type": all_models}
    data["pytorch"] = [pt_models[t] for t in all_models]
    data["tensorflow"] = [tf_models[t] for t in all_models]
    data["flax"] = [flax_models[t] for t in all_models]

    # Now let's find the right processing class for each model. In order we check if there is a Processor, then a
    # Tokenizer, then a FeatureExtractor, then an ImageProcessor
    processors = {}
    for t in all_models:
        if t in transformers_module.models.auto.processing_auto.PROCESSOR_MAPPING_NAMES:
            processors[t] = "AutoProcessor"
        elif t in transformers_module.models.auto.tokenization_auto.TOKENIZER_MAPPING_NAMES:
            processors[t] = "AutoTokenizer"
        elif t in transformers_module.models.auto.image_processing_auto.IMAGE_PROCESSOR_MAPPING_NAMES:
            processors[t] = "AutoImageProcessor"
        elif t in transformers_module.models.auto.feature_extraction_auto.FEATURE_EXTRACTOR_MAPPING_NAMES:
            processors[t] = "AutoFeatureExtractor"
        else:
            # Default to AutoTokenizer if a model has nothing, for backward compatibility.
            processors[t] = "AutoTokenizer"

    data["processor"] = [processors[t] for t in all_models]



SUMMARY:
**Function Summary: `get_frameworks_table`**

**Primary Purpose:** Generates a pandas DataFrame containing the supported auto classes for each model type, using the content of the auto modules.

**Key Implementation Details:**

* Uses two dictionaries to map model prefixes to model types (`model_prefix_to_model_type`)
* Populates three separate dictionaries (`pt_models`, `tf_models`, and `flax_models`) to track backend support for each model prefix
* Iterates through all transformers object attributes using regular expressions to identify models supported by PT, TF, or Flax backends

**Notable Dependencies and Requirements:**

* `pd` (pandas library) for data manipulation
* `transformers_module` (external dependencies) containing configuration mapping names and processing class mappings
* `_re_tf_models`, `_re_flax_models`, and `_re_pt_models` (regex patterns used to identify models supported by PT, TF, or Flax backends)
* `collections` (utility module) for using defaultdicts


key: chroma:document
string_value:
CODE:

def update_pipeline_and_auto_class_table(table: Dict[str, Tuple[str, str]]) -> Dict[str, Tuple[str, str]]:
    """
    Update the table maping models to pipelines and auto classes without removing old keys if they don't exist anymore.

    Args:
        table (`Dict[str, Tuple[str, str]]`):
            The existing table mapping model names to a tuple containing the pipeline tag and the auto-class name with
            which they should be used.

    Returns:
        `Dict[str, Tuple[str, str]]`: The updated table in the same format.
    """
    auto_modules = [
        transformers_module.models.auto.modeling_auto,
        transformers_module.models.auto.modeling_tf_auto,
        transformers_module.models.auto.modeling_flax_auto,
    ]
    for pipeline_tag, model_mapping, auto_class in PIPELINE_TAGS_AND_AUTO_MODELS:
        model_mappings = [model_mapping, f"TF_{model_mapping}", f"FLAX_{model_mapping}"]
        auto_classes = [auto_class, f"TF_{auto_class}", f"Flax_{auto_class}"]
        # Loop through all three frameworks
        for module, cls, mapping in zip(auto_modules, auto_classes, model_mappings):
            # The type of pipeline may not exist in this framework
            if not hasattr(module, mapping):
                continue
            # First extract all model_names
            model_names = []
            for name in getattr(module, mapping).values():
                if isinstance(name, str):
                    model_names.append(name)
                else:
                    model_names.extend(list(name))

            # Add pipeline tag and auto model class for those models
            table.update({model_name: (pipeline_tag, cls) for model_name in model_names})



SUMMARY:
**Function Summary**

### `update_pipeline_and_auto_class_table`

Updates the table mapping models to pipelines and auto classes without removing old keys.

#### Parameters
- `table`: The existing table mapping model names to a tuple containing the pipeline tag and the auto-class name (Dict[str, Tuple[str, str]])

#### Returns
- `Dict[str, Tuple[str, str]]`: The updated table in the same format

#### Implementation Details

- Updates table with pipeline tags and auto-model classes for each framework (transformers_module.models.auto.modeling_auto, transformers_module.models.auto.modeling_tf_auto, transformers_module.models.auto.modeling_flax_auto)
- Handles cases where pipeline type may not exist in a given framework
- Uses model mappings and pipepline tags to populate the table

#### Notable Dependencies
- `PIPELINE_TAGS_AND_AUTO_MODELS`
- `transformers_module.models.auto.modeling_auto`, `transformers_module.models.auto.modeling_tf_auto`, `transformers_module.models.auto.modeling_flax_auto`

#### External Dependencies
- `transformers_module`

#### Notes

This function iterates over a list of frameworks, extracts model names, and updates the table with pipeline tags and auto-model classes.


key: chroma:document
string_value:
CODE:

def update_metadata(token: str, commit_sha: str):
    """
    Update the metadata for the Transformers repo in `huggingface/transformers-metadata`.

    Args:
        token (`str`): A valid token giving write access to `huggingface/transformers-metadata`.
        commit_sha (`str`): The commit SHA on Transformers corresponding to this update.
    """
    frameworks_table = get_frameworks_table()
    frameworks_dataset = Dataset.from_pandas(frameworks_table)

    resolved_tags_file = hf_hub_download(
        "huggingface/transformers-metadata", "pipeline_tags.json", repo_type="dataset", token=token
    )
    tags_dataset = Dataset.from_json(resolved_tags_file)
    table = {
        tags_dataset[i]["model_class"]: (tags_dataset[i]["pipeline_tag"], tags_dataset[i]["auto_class"])
        for i in range(len(tags_dataset))
    }
    table = update_pipeline_and_auto_class_table(table)

    # Sort the model classes to avoid some nondeterministic updates to create false update commits.
    model_classes = sorted(table.keys())
    tags_table = pd.DataFrame(
        {
            "model_class": model_classes,
            "pipeline_tag": [table[m][0] for m in model_classes],
            "auto_class": [table[m][1] for m in model_classes],
        }
    )
    tags_dataset = Dataset.from_pandas(tags_table)

    hub_frameworks_json = hf_hub_download(
        repo_id="huggingface/transformers-metadata",
        filename="frameworks.json",
        repo_type="dataset",
        token=token,
    )
    with open(hub_frameworks_json) as f:
        hub_frameworks_json = f.read()

    hub_pipeline_tags_json = hf_hub_download(
        repo_id="huggingface/transformers-metadata",
        filename="pipeline_tags.json",
        repo_type="dataset",
        token=token,
    )
    with open(hub_pipeline_tags_json) as f:
        hub_pipeline_tags_json = f.read()

    with tempfile.TemporaryDirectory() as tmp_dir:
        frameworks_dataset.to_json(os.path.join(tmp_dir, "frameworks.json"))
        tags_dataset.to_json(os.path.join(tmp_dir, "pipeline_tags.json"))

        with open(os.path.join(tmp_dir, "frameworks.json")) as f:
            frameworks_json = f.read()
        with open(os.path.join(tmp_dir, "pipeline_tags.json")) as f:
            pipeline_tags_json = f.read()

        frameworks_equal = hub_frameworks_json == frameworks_json
        hub_pipeline_tags_equal = hub_pipeline_tags_json == pipeline_tags_json

        if frameworks_equal and hub_pipeline_tags_equal:
            print("No updates on the Hub, not pushing the metadata files.")
            return

        if commit_sha is not None:
            commit_message = (
                f"Update with commit {commit_sha}\n\nSee: "
                f"https://github.com/huggingface/transformers/commit/{commit_sha}"
            )
        else:
            commit_message = "Update"

        upload_folder(
            repo_id="huggingface/transformers-metadata",
            folder_path=tmp_dir,
            repo_type="dataset",
            token=token,
            commit_message=commit_message,


SUMMARY:
**Function Summary:**

`update_metadata`

**Purpose:** Updates the metadata for the Transformers repository by processing and uploading dataset files.

**Key Implementation Details:**

* Downloads pipeline_tags.json and frameworks.json dataset files from huggingface/transformers-metadata
* Processes framework and tag data from downloaded files
* Checks if updates need to be pushed to GitHub and commits message is generated accordingly

**Noteable Dependencies/Requirements:**

* `get_frameworks_table` function (not shown in this code snippet)
* `Dataset.from_json` and `Dataset.from_pandas` functions (from the Hugging Face datasets library)
* `hf_hub_download` and `upload_folder` functions (from the Hugging Face Hub client library)

**Interfaces/ Method Signatures:**

* `update_pipeline_and_auto_class_table(table)`: Updates table with processed framework data
* `get_frameworks_table()`: Returns a pandas dataframe containing frameworks data

**External Dependencies:**

* `huggingface/transformers-metadata`
* `huggingfacedatasets` library
* `huggingface_hub` library


key: chroma:document
string_value:
CODE:

def check_pipeline_tags():
    """
    Check all pipeline tags are properly defined in the `PIPELINE_TAGS_AND_AUTO_MODELS` constant of this script.
    """
    in_table = {tag: cls for tag, _, cls in PIPELINE_TAGS_AND_AUTO_MODELS}
    pipeline_tasks = transformers_module.pipelines.SUPPORTED_TASKS
    missing = []
    for key in pipeline_tasks:
        if key not in in_table:
            model = pipeline_tasks[key]["pt"]
            if isinstance(model, (list, tuple)):
                model = model[0]
            model = model.__name__
            if model not in in_table.values():
                missing.append(key)

    if len(missing) > 0:
        msg = ", ".join(missing)
        raise ValueError(
            "The following pipeline tags are not present in the `PIPELINE_TAGS_AND_AUTO_MODELS` constant inside "
            f"`utils/update_metadata.py`: {msg}. Please add them!"


SUMMARY:
**Summary:**

This function, `check_pipeline_tags()`, detects and raises an error if any pipeline tasks are missing from the predefined `PIPELINE_TAGS_AND_AUTO_MODELS` constant.

**Key Details:**

- Verifies that all task pipelines match the predefined auto-models.
- Checks against `transformers_module.pipelines.SUPPORTED_TASKS`.
- Raises a `ValueError` with missing task pipelines as input when not found in the defined constants.

**Implementation Notes:**

* Utilizes dictionary comprehension to create an in-table mapping of pipeline tags and models.
* Iterates through supported tasks, checking their existence in the predefined constant map (`in_table`).
* Raises an error if the `missing` list is populated during iteration. 

This function ensures that all defined pipelines have corresponding auto-models.


key: docstring
string_value:
r"""
The argument `test_file` in this file refers to a model test file. This should be a string of the from
`tests/models/*/test_modeling_*.py`.
"""


key: chroma:document
string_value:
CODE:

def get_module_path(test_file):
    """Return the module path of a model test file."""
    components = test_file.split(os.path.sep)
    if components[0:2] != ["tests", "models"]:
        raise ValueError(
            "`test_file` should start with `tests/models/` (with `/` being the OS specific path separator). Got "
            f"{test_file} instead."
        )
    test_fn = components[-1]
    if not test_fn.endswith("py"):
        raise ValueError(f"`test_file` should be a python file. Got {test_fn} instead.")
    if not test_fn.startswith("test_modeling_"):
        raise ValueError(
            f"`test_file` should point to a file name of the form `test_modeling_*.py`. Got {test_fn} instead."
        )

    components = components[:-1] + [test_fn.replace(".py", "")]
    test_module_path = ".".join(components)



SUMMARY:
**Function Summary**

The `get_module_path` function takes a `test_file` path as input and returns the corresponding module path. It validates that the input path matches the expected format `tests/modeling/*.py`, raising errors for any deviations.

**Key Implementation Details**

- Splits the input `test_file` into components using the OS-specific path separator.
- Verifies that the first two components are "tests" and "models," and the last component ends with ".py."
- Reconstructs the module path from the validated components.

**Notable Dependencies and Requirements**

- Uses the `os` module for handling file paths.
- References `python files` directly in validation checks.


key: chroma:document
string_value:
CODE:

def get_test_module(test_file):
    """Get the module of a model test file."""
    test_module_path = get_module_path(test_file)
    try:
        test_module = importlib.import_module(test_module_path)
    except AttributeError as exc:
        # e.g. if you have a `tests` folder in `site-packages`, created by another package, when trying to import
        # `tests.models...`
        raise ValueError(
            f"Could not import module {test_module_path}. Confirm that you don't have a package with the same root "
            "name installed or in your environment's `site-packages`."
        ) from exc



SUMMARY:
**Summary:**

This function, `get_test_module`, retrieves and loads the Python module corresponding to a given test file by its path. It identifies if the import is successful or not and provides an informative error message if an AttributeError occurs.

**Key Implementation Details:**

* Imports a module using `importlib.import_module` with a specified path.
* Raises a ValueError with an informative message if the import fails due to a common issue.

**Notable Dependencies:**

* `importlib`
* None

**Interfaces/Classes:**

* No external interface or class is used in this function.


key: chroma:document
string_value:
CODE:

def get_tester_classes(test_file):
    """Get all classes in a model test file whose names ends with `ModelTester`."""
    tester_classes = []
    test_module = get_test_module(test_file)
    for attr in dir(test_module):
        if attr.endswith("ModelTester"):
            tester_classes.append(getattr(test_module, attr))

    # sort with class names


SUMMARY:
**Code Summary**

### Purpose and Functionality

Get all classes in a model test file whose names end with "ModelTester".

### Implementation Details

* Uses the `dir()` function to list attributes of the `test_module` object.
* Filters attributes based on name ending with "ModelTester".
* Retrieves attribute objects using `getattr()`.
* Sorts retrieved classes by their names using the `sorted()` function and a lambda function.

### Notable Dependencies or Requirements

* `get_test_module`: Returns the test module for the given test file (not shown in this code snippet).
 
This summary preserves detailed information from the original code while minimizing unnecessary details.


key: chroma:document
string_value:
CODE:

def get_test_classes(test_file):
    """Get all [test] classes in a model test file with attribute `all_model_classes` that are non-empty.

    These are usually the (model) test classes containing the (non-slow) tests to run and are subclasses of one of the
    classes `ModelTesterMixin`, `TFModelTesterMixin` or `FlaxModelTesterMixin`, as well as a subclass of
    `unittest.TestCase`. Exceptions include `RagTestMixin` (and its subclasses).
    """
    test_classes = []
    test_module = get_test_module(test_file)
    for attr in dir(test_module):
        attr_value = getattr(test_module, attr)
        # (TF/Flax)ModelTesterMixin is also an attribute in specific model test module. Let's exclude them by checking
        # `all_model_classes` is not empty (which also excludes other special classes).
        model_classes = getattr(attr_value, "all_model_classes", [])
        if len(model_classes) > 0:
            test_classes.append(attr_value)

    # sort with class names


SUMMARY:
Here is a concise summary of the code chunk:

**Function Purpose**
The `get_test_classes` function retrieves and sorts a list of non-empty test classes in a model test file that inherit from specific mixins (`ModelTesterMixin`, `TFModelTesterMixin`, or `FlaxModelTesterMixin`) or `unittest.TestCase`.

**Key Implementation Details**

* Iterates through the module's attributes to find classes with an empty `all_model_classes` attribute.
* Excludes exceptions like `RagTestMixin` and its subclasses.
* Sorts the test classes by their names using `__name__` attribute.

**Notable Dependencies and Requirements**

* The `get_test_module` function is called from within this function, suggesting a dependency on another module (`test_file`).
* This function relies on `unittest.TestCase`, indicating a requirement for testing framework.


key: chroma:document
string_value:
CODE:

def get_model_classes(test_file):
    """Get all model classes that appear in `all_model_classes` attributes in a model test file."""
    test_classes = get_test_classes(test_file)
    model_classes = set()
    for test_class in test_classes:
        model_classes.update(test_class.all_model_classes)

    # sort with class names


SUMMARY:
**Function Summary**

`get_model_classes`

Returns a sorted list of unique model classes from `all_model_classes` attributes in a model test file.

**Key Implementation Details**

* Uses `set` to eliminate duplicate class instances
* Iterates over `test_classes`, updating `model_classes` with `all_model_classes`
* Sorts classes by name using a case-insensitive comparison

**Notable Dependencies and Requirements**

* `get_test_classes` function is called with `test_file` as an argument
* Uses `__name__` attribute of class names for sorting comparison
* No external dependencies are listed due to lack of information


key: chroma:document
string_value:
CODE:

def get_model_tester_from_test_class(test_class):
    """Get the model tester class of a model test class."""
    test = test_class()
    if hasattr(test, "setUp"):
        test.setUp()

    model_tester = None
    if hasattr(test, "model_tester"):
        # `(TF/Flax)ModelTesterMixin` has this attribute default to `None`. Let's skip this case.
        if test.model_tester is not None:
            model_tester = test.model_tester.__class__



SUMMARY:
**Summary**

The `get_model_tester_from_test_class` function takes a `test_class` as input and returns the model tester class associated with it. It checks for the presence of an instance with an explicit `setUp` method, then attempts to retrieve the model tester class from the instance's attributes.

**Key Implementation Details**

* Tries to call `setUp` on the test instance if present
* Ignores cases where `model_tester` is set to `None`, likely due to default values in `ModelTesterMixin`

**Notable Dependencies/Requirements**

* `test_class`: The input class being analyzed
* `TF/Flax` (implied): The framework used for the `ModelTesterMixin`
* No external dependencies noted

**Notes**: This code appears to be part of a testing harness, likely for machine learning models.


key: chroma:document
string_value:
CODE:

def get_test_classes_for_model(test_file, model_class):
    """Get all [test] classes in `test_file` that have `model_class` in their `all_model_classes`."""
    test_classes = get_test_classes(test_file)

    target_test_classes = []
    for test_class in test_classes:
        if model_class in test_class.all_model_classes:
            target_test_classes.append(test_class)

    # sort with class names


SUMMARY:
**Function Summary**

`get_test_classes_for_model`: Returns a list of test classes from `test_file` that include `model_class`. 

**Key Implementation Details**

* Uses `sorted` function for alphabetical ordering by class name (`x.__name__`)
* Iterates through all test classes using `get_test_classes(test_file)`
* Filters test classes based on presence of `model_class` in `all_model_classes`

**Input/Output**

* `test_file`: input file containing test classes
* `model_class`: model class to search for in test classes
* Returns a sorted list of matching test classes

**Dependencies/Requirements**

* `get_test_classes` function from same library or module


key: chroma:document
string_value:
CODE:

def get_tester_classes_for_model(test_file, model_class):
    """Get all model tester classes in `test_file` that are associated to `model_class`."""
    test_classes = get_test_classes_for_model(test_file, model_class)

    tester_classes = []
    for test_class in test_classes:
        tester_class = get_model_tester_from_test_class(test_class)
        if tester_class is not None:
            tester_classes.append(tester_class)

    # sort with class names


SUMMARY:
**Summary**

Returns a list of model tester classes from a test file (`test_file`) associated with the specified `model_class`, sorted alphabetically by class name.

**Key Details**

* **Functionality**: Retrieves and filters model tester classes based on the provided `test_file` and `model_class`.
* **Algorithm/Design Choice**: Sorts filtered tester classes using the `__name__` attribute.
* **Method Signatures**:
	+ `get_test_classes_for_model()`: returns list of test classes for a given model
	+ `get_model_tester_from_test_class(test_class)`: returns associated model tester class from a test class

**Dependencies**

* `get_test_classes_for_model`
* `get_model_tester_from_test_class`


key: chroma:document
string_value:
CODE:

def get_test_to_tester_mapping(test_file):
    """Get a mapping from [test] classes to model tester classes in `test_file`.

    This uses `get_test_classes` which may return classes that are NOT subclasses of `unittest.TestCase`.
    """
    test_classes = get_test_classes(test_file)
    test_tester_mapping = {test_class: get_model_tester_from_test_class(test_class) for test_class in test_classes}


SUMMARY:
**Summary**

`get_test_to_tester_mapping` function:
 
* Maps test classes to model tester classes based on inheritance from `unittest.TestCase`.
* Uses `get_test_classes` to retrieve test class list and `get_model_tester_from_test_class` to map each test class.

**Key Details:**

* Reusable in a unit testing framework.
* May return classes that are not subclasses of `unittest.TestCase`.

**Implementation Notes:**

* Uses dictionary comprehension for efficient creation of mapping.
* Relies on `get_test_classes` and `get_model_tester_from_test_class` external functions.


key: chroma:document
string_value:
CODE:

def get_model_to_test_mapping(test_file):
    """Get a mapping from model classes to test classes in `test_file`."""
    model_classes = get_model_classes(test_file)
    model_test_mapping = {
        model_class: get_test_classes_for_model(test_file, model_class) for model_class in model_classes
    }


SUMMARY:
**Summary:**

`get_model_to_test_mapping` function generates a dictionary mapping `model_classes` to their corresponding test classes (`test_classes`) based on the input `test_file`. It uses `get_model_classes` and `get_test_classes_for_model` functions.

**Key Details:**

- Uses `get_model_classes(test_file)` to retrieve model class names
- Populates a dictionary (`model_test_mapping`) with models as keys and test classes as values using map comprehension

**Notable Dependencies/Requirements:**

- `get_model_classes(test_file)`
- `get_test_classes_for_model(test_file, model_class)`

It can be summarized as:
```python
def get_model_to_test_mapping(test_file):
    # Maps model classes to their corresponding test classes
    model_classes = get_model_classes(test_file)
    model_test_mapping = {model_class: get_test_classes_for_model(test_file, model_class) for model_class in model_classes}
    return model_test_mapping
```


key: chroma:document
string_value:
CODE:

def get_model_to_tester_mapping(test_file):
    """Get a mapping from model classes to model tester classes in `test_file`."""
    model_classes = get_model_classes(test_file)
    model_to_tester_mapping = {
        model_class: get_tester_classes_for_model(test_file, model_class) for model_class in model_classes
    }


SUMMARY:
**Summary**

`get_model_to_tester_mapping` function maps model classes to their corresponding tester classes in a given test file.

**Key Details:**

* Uses `model_classes` from `get_model_classes` for iteration
* Iterates using dictionary comprehension to create mappings of model classes to tester classes
* Returns a dictionary `model_to_tester_mapping`

**Notable Dependencies and Interfaces:**

* References `get_model_classes` and `get_tester_classes_for_model` functions (not shown)


key: chroma:document
string_value:
CODE:

def to_json(o):
    """Make the information succinct and easy to read.

    Avoid the full class representation like `<class 'transformers.models.bert.modeling_bert.BertForMaskedLM'>` when
    displaying the results. Instead, we use class name (`BertForMaskedLM`) for the readability.
    """
    if isinstance(o, str):
        return o
    elif isinstance(o, type):
        return o.__name__
    elif isinstance(o, (list, tuple)):
        return [to_json(x) for x in o]
    elif isinstance(o, dict):
        return {to_json(k): to_json(v) for k, v in o.items()}
    else:


SUMMARY:
**Concise Summary**

### Function Purpose and Main Functionality
The `to_json` function converts an object `o` into a human-readable JSON format. It handles various data types by recursively processing nested structures, truncating class representations for better readability.

### Key Implementation Details

* Handles string, type alias, list, tuple, and dictionary conversions using recursive calls.
* Uses `__name__` attribute to display only class names instead of full class representations.

### Notable Dependencies or Requirements
None mentioned, but it assumes objects are usable in a context that supports nested data processing (e.g., lists, dictionaries).

### Key Method Signatures and Interfaces

* Implicit: No explicit interfaces defined.
* Explicit: None.


key: chroma:document
string_value:
CODE:

def get_checkpoint_from_config_class(config_class):
    checkpoint = None

    # source code of `config_class`
    config_source = inspect.getsource(config_class)
    checkpoints = _re_checkpoint.findall(config_source)

    # Each `checkpoint` is a tuple of a checkpoint name and a checkpoint link.
    # For example, `('google-bert/bert-base-uncased', 'https://huggingface.co/google-bert/bert-base-uncased')`
    for ckpt_name, ckpt_link in checkpoints:
        # allow the link to end with `/`
        if ckpt_link.endswith("/"):
            ckpt_link = ckpt_link[:-1]

        # verify the checkpoint name corresponds to the checkpoint link
        ckpt_link_from_name = f"https://huggingface.co/{ckpt_name}"
        if ckpt_link == ckpt_link_from_name:
            checkpoint = ckpt_name
            break



SUMMARY:
**Summary:**

`get_checkpoint_from_config_class` function retrieves the corresponding checkpoint name from a provided config class's source code.

*   **Purpose:** To extract the checkpoint name from a config class based on its implementation details.
*   **Key Details:**
    *   Uses regular expression to find checkpoints in the config class source code.
    *   Verifies each checkpoint by comparing its name with the corresponding link.
*   **Notable Dependencies/Requirements:**
    *   `_re_checkpoint` regular expression pattern
    *   `inspect.getsource(config_class)` function
*   **Interfaces/Methods:** None (uses built-in Python functions and modules)
*   **External Dependencies:**

    *   `_re_checkpoint`
    *   `config_class` (input parameter)

**Summary Focus Criteria Met:**

1. Precise technical terms used: "config class", "checkpoint name", "regular expression pattern"
2. Class/function/variable names preserved exactly.
3. Primary purpose stated: "To extract the checkpoint name from a config class based on its implementation details."
4. Key implementation details narrowed down to:
    *   Regular expression usage
    *   Comparison verification for checkpoints
5. Notable dependencies listed with exact class names and functions: `_re_checkpoint`, `config_class`.
6. No external modules or libraries required besides built-in Python functions.


key: chroma:document
string_value:
CODE:

def check_config_docstrings_have_checkpoints():
    configs_without_checkpoint = []

    for config_class in list(CONFIG_MAPPING.values()):
        # Skip deprecated models
        if "models.deprecated" in config_class.__module__:
            continue
        checkpoint = get_checkpoint_from_config_class(config_class)

        name = config_class.__name__
        if checkpoint is None and name not in CONFIG_CLASSES_TO_IGNORE_FOR_DOCSTRING_CHECKPOINT_CHECK:
            configs_without_checkpoint.append(name)

    if len(configs_without_checkpoint) > 0:
        message = "\n".join(sorted(configs_without_checkpoint))
        raise ValueError(
            f"The following configurations don't contain any valid checkpoint:\n{message}\n\n"
            "The requirement is to include a link pointing to one of the models of this architecture in the "
            "docstring of the config classes listed above. The link should have be a markdown format like "
            "[myorg/mymodel](https://huggingface.co/myorg/mymodel)."


SUMMARY:
**Function Summary**

`check_config_docstrings_have_checkpoints()`: Verifies if all configuration classes in `CONFIG_MAPPING` have valid checkpoints in their docstrings. Raises a ValueError if any configurations lack checkpoint information.

**Key Implementation Details**

* Skips deprecated models by checking for "models.deprecated" in the module name.
* Iterates over config classes in `CONFIG_MAPPING`, appending names of missing checkpoint info to `configs_without_checkpoint` list.
* If no checkpoints found, raises a ValueError with an error message and sorted list of configuration class names.

**Notable Dependencies and Requirements**

* `CONFIG_MAPPING`: mapping of configuration classes
* `CONFIG_CLASSES_TO_IGNORE_FOR_DOCSTRING_CHECKPOINT_CHECK`: list of ignored configuration class names for docstring checkpoint check
* `get_checkpoint_from_config_class(config_class)`: function to retrieve checkpoint from a config class
* `sorted()` function for sorting configuration class names.


key: chroma:document
string_value:
CODE:

def handle_test_results(test_results):
    expressions = test_results.split(" ")

    failed = 0
    success = 0

    # When the output is short enough, the output is surrounded by = signs: "== OUTPUT =="
    # When it is too long, those signs are not present.
    time_spent = expressions[-2] if "=" in expressions[-1] else expressions[-1]

    for i, expression in enumerate(expressions):
        if "failed" in expression:
            failed += int(expressions[i - 1])
        if "passed" in expression:
            success += int(expressions[i - 1])



SUMMARY:
**Summary**

The `handle_test_results` function parses test results to extract failure and success counts, as well as the timestamp. Key aspects include:

*   **Purpose**: Parse test output to calculate failure and success metrics.
*   **Implementation Details**:
    *   Splits test output into expressions using spaces as separators.
    *   Calculates `time_spent` by detecting presence of `== OUTPUT ==` signs in the last expression.
    *   Iterates through expressions, incrementing `failed` and `success` counts based on specific checks (e.g., "failed" or "passed").
*   **Notable Dependencies/Requirements**:
    *   None

**Detailed Information**

*   Method Signatures: Not applicable
*   Interfaces Implemented: None
*   External Dependencies: None


key: chroma:document
string_value:
CODE:

def extract_first_line_failure(failures_short_lines):
    failures = {}
    file = None
    in_error = False
    for line in failures_short_lines.split("\n"):
        if re.search(r"_ \[doctest\]", line):
            in_error = True
            file = line.split(" ")[2]
        elif in_error and not line.split(" ")[0].isdigit():
            failures[file] = line
            in_error = False



SUMMARY:
**Summary:**

This function, `extract_first_line_failure`, processes text data from "short lines" of failures. It categorizes them into file-related failures by identifying the `_ [doctest]` prefix and extracts specific information about the error.

**Key Details:**

- **Algorithm:** Uses regular expressions to scan short lines for a pattern (_ [doctest]).
- **Method Signatures/Interfaces:** Takes `failures_short_lines` as input and returns a dictionary of failures (`failures`) with file names.
- **Dependencies:**
  - Utilizes external `re` (regular expression) module.
  - No imported classes/functions besides built-in Python.

**Context:**
This function appears to be part of a code analysis or testing framework, likely used for data extraction from test output logs.


key: chroma:document
string_value:
CODE:

class Message:
    def __init__(self, title: str, doc_test_results: Dict):
        self.title = title

        self.n_success = sum(job_result["n_success"] for job_result in doc_test_results.values())
        self.n_failures = sum(job_result["n_failures"] for job_result in doc_test_results.values())
        self.n_tests = self.n_success + self.n_failures

        # Failures and success of the modeling tests
        self.doc_test_results = doc_test_results

    @property
    def time(self) -> str:
        all_results = [*self.doc_test_results.values()]
        time_spent = [r["time_spent"].split(", ")[0] for r in all_results if len(r["time_spent"])]
        total_secs = 0

        for time in time_spent:
            time_parts = time.split(":")

            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.
            if len(time_parts) == 1:
                time_parts = [0, 0, time_parts[0]]

            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])
            total_secs += hours * 3600 + minutes * 60 + seconds

        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60
        return f"{int(hours)}h{int(minutes)}m{int(seconds)}s"

    @property
    def header(self) -> Dict:
        return {"type": "header", "text": {"type": "plain_text", "text": self.title}}

    @property
    def no_failures(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": f"🌞 There were no failures: all {self.n_tests} tests passed. The suite ran in {self.time}.",
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def failures(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": (
                    f"There were {self.n_failures} failures, out of {self.n_tests} tests.\nThe suite ran in"
                    f" {self.time}."
                ),
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def category_failures(self) -> List[Dict]:
        failure_blocks = []

        MAX_ERROR_TEXT = 3000 - len("The following examples had failures:\n\n\n\n") - len("[Truncated]\n")
        line_length = 40
        category_failures = {k: v["failed"] for k, v in doc_test_results.items() if isinstance(v, dict)}

        def single_category_failures(category, failures):
            text = ""
            if len(failures) == 0:
                return ""
            text += f"*{category} failures*:".ljust(line_length // 2).rjust(line_length // 2) + "\n"

            for idx, failure in enumerate(failures):
                new_text = text + f"`{failure}`\n"
                if len(new_text) > MAX_ERROR_TEXT:
                    text = text + "[Truncated]\n"
                    break
                text = new_text

            return text

        for category, failures in category_failures.items():
            report = single_category_failures(category, failures)
            if len(report) == 0:
                continue
            block = {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": f"The following examples had failures:\n\n\n{report}\n",
                },
            }
            failure_blocks.append(block)

        return failure_blocks

    @property
    def payload(self) -> str:
        blocks = [self.header]

        if self.n_failures > 0:
            blocks.append(self.failures)

        if self.n_failures > 0:
            blocks.extend(self.category_failures)

        if self.n_failures == 0:
            blocks.append(self.no_failures)

        return json.dumps(blocks)

    @staticmethod
    def error_out():
        payload = [
            {
                "type": "section",
                "text": {
                    "type": "plain_text",
                    "text": "There was an issue running the tests.",
                },
                "accessory": {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                    "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
                },
            }
        ]

        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(payload)}))

        client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            text="There was an issue running the tests.",
            blocks=payload,
        )

    def post(self):
        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(self.payload)}))

        text = f"{self.n_failures} failures out of {self.n_tests} tests," if self.n_failures else "All tests passed."

        self.thread_ts = client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            blocks=self.payload,
            text=text,
        )

    def get_reply_blocks(self, job_name, job_link, failures, text):
        # `text` must be less than 3001 characters in Slack SDK
        # keep some room for adding "[Truncated]" when necessary
        MAX_ERROR_TEXT = 3000 - len("[Truncated]")

        failure_text = ""
        for key, value in failures.items():
            new_text = failure_text + f"*{key}*\n_{value}_\n\n"
            if len(new_text) > MAX_ERROR_TEXT:
                # `failure_text` here has length <= 3000
                failure_text = failure_text + "[Truncated]"
                break
            # `failure_text` here has length <= MAX_ERROR_TEXT
            failure_text = new_text

        title = job_name
        content = {"type": "section", "text": {"type": "mrkdwn", "text": text}}

        if job_link is not None:
            content["accessory"] = {
                "type": "button",
                "text": {"type": "plain_text", "text": "GitHub Action job", "emoji": True},
                "url": job_link,
            }

        return [
            {"type": "header", "text": {"type": "plain_text", "text": title, "emoji": True}},
            content,
            {"type": "section", "text": {"type": "mrkdwn", "text": failure_text}},
        ]

    def post_reply(self):
        if self.thread_ts is None:
            raise ValueError("Can only post reply if a post has been made.")

        sorted_dict = sorted(self.doc_test_results.items(), key=lambda t: t[0])
        for job_name, job_result in sorted_dict:
            if len(job_result["failures"]) > 0:
                text = f"*Num failures* :{len(job_result['failed'])} \n"
                failures = job_result["failures"]
                blocks = self.get_reply_blocks(job_name, job_result["job_link"], failures, text=text)

                print("Sending the following reply")
                print(json.dumps({"blocks": blocks}))

                client.chat_postMessage(
                    channel=SLACK_REPORT_CHANNEL_ID,
                    text=f"Results for {job_name}",
                    blocks=blocks,
                    thread_ts=self.thread_ts["ts"],
                )



SUMMARY:
**Class Summary:**

The `Message` class is designed to represent test results and report them in a Slack channel. It has the following primary purposes:

*   Calculate and store metadata about the test results (e.g., number of successes/failures, time spent on tests)
*   Generate reports with various information types (header, failures by category, payload)

**Notable Properties and Methods:**

*   `n_failures`, `n_tests`: Number of test failures and total tests
*   `payload`: A Slack-compatible report payload in JSON format

Key methods:

*   `post()`: Posts the test result report to the specified Slack channel.
*   `error_out()`: Sends a notification with an error message to the same Slack channel.
*   `get_reply_blocks()`: Generates reply blocks for individual test failures, including links to GitHub Action jobs.


key: chroma:document
string_value:
CODE:

def retrieve_artifact(name: str):
    _artifact = {}

    if os.path.exists(name):
        files = os.listdir(name)
        for file in files:
            try:
                with open(os.path.join(name, file), encoding="utf-8") as f:
                    _artifact[file.split(".")[0]] = f.read()
            except UnicodeDecodeError as e:
                raise ValueError(f"Could not open {os.path.join(name, file)}.") from e



SUMMARY:
**Summary**

`retrieve_artifact` function retrieves files from a specified directory and returns an artifact dictionary containing file names as keys and their corresponding contents.

**Main Purpose and Functionality**
The primary purpose of this function is to read the contents files from a given directory named `name`. The function iterates over each file, attempts to decode its contents, and stores them in a dictionary.

```python
def retrieve_artifact(name: str) -> dict:
    # ... (function implementation remains the same)
```

**Key Implementation Details**

* Uses `os.path.exists` to check if the specified directory path exists.
* Lists files using `os.listdir`.
* Tries to open and read each file with UTF-8 encoding, raising an error for decoding failures.

```python
files = os.listdir(name)
for file in files:
    try:
        with open(os.path.join(name, file), encoding="utf-8") as f:
            _artifact[file.split(".")[0]] = f.read()
```

**Notable Dependencies and Requirements**

* `os` module for path manipulation.
* External dependency: `file extensions (e.g., `.txt`, `.pdf`).
* No apparent inheritance or interface implementation.

```python
import os
```

**RAG System Representation**

| Key | Value |
| --- | --- |
| Purpose | Retrieve files from a specified directory and return an artifact dictionary. |
| Functionality | Iterates over each file, attempts to decode its contents, and stores them in a dictionary. |
| Implementation Details | Uses `os.path.exists`, `os.listdir`, and UTF-8 encoding for file reading. |
| Dependencies | `os` module. |


key: chroma:document
string_value:
CODE:

def retrieve_available_artifacts():
    class Artifact:
        def __init__(self, name: str):
            self.name = name
            self.paths = []

        def __str__(self):
            return self.name

        def add_path(self, path: str):
            self.paths.append({"name": self.name, "path": path})

    _available_artifacts: Dict[str, Artifact] = {}

    directories = filter(os.path.isdir, os.listdir())
    for directory in directories:
        artifact_name = directory
        if artifact_name not in _available_artifacts:
            _available_artifacts[artifact_name] = Artifact(artifact_name)

            _available_artifacts[artifact_name].add_path(directory)



SUMMARY:
**Concise Summary:**

This function, `retrieve_available_artifacts`, creates and populates a dictionary of artifacts stored in directories on the system. It utilizes classes to encapsulate artifact details and behavior.

**Key Implementation Details:**

* Creates an `Artifact` class with `name` attribute, which stores directory names as names and their corresponding paths.
* Utilizes `filter()` and `os.path.isdir()` to retrieve only directories from the system.
* Popsulates `_available_artifacts` dictionary with the extracted directory information.

**Notable Dependencies:**

* `os`: for interacting with the operating system (e.g., `os.path.isdir()`)
* `filter()`: a built-in Python function for filtering elements from iterable collections based on condition.

**Primary Purpose and Functionality:**
The main purpose of this code is to retrieve available artifacts stored in directories, which can be utilized for subsequent operations or processing. 

**Additional Informed Insights:**
- The code iterates over all files and folders in the system using `os.listdir()`.
- It populates an internal data structure `_available_artifacts` dictionary, containing only directories as keys.
- It assumes that the directory names match the artifact names within these instances, allowing for consistent categorization.


key: chroma:document
string_value:
CODE:

def clean_model_doc_toc(model_doc: List[dict]) -> List[dict]:
    """
    Cleans a section of the table of content of the model documentation (one specific modality) by removing duplicates
    and sorting models alphabetically.

    Args:
        model_doc (`List[dict]`):
            The list of dictionaries extracted from the `_toctree.yml` file for this specific modality.

    Returns:
        `List[dict]`: List of dictionaries like the input, but cleaned up and sorted.
    """
    counts = defaultdict(int)
    for doc in model_doc:
        counts[doc["local"]] += 1
    duplicates = [key for key, value in counts.items() if value > 1]

    new_doc = []
    for duplicate_key in duplicates:
        titles = list({doc["title"] for doc in model_doc if doc["local"] == duplicate_key})
        if len(titles) > 1:
            raise ValueError(
                f"{duplicate_key} is present several times in the documentation table of content at "
                "`docs/source/en/_toctree.yml` with different *Title* values. Choose one of those and remove the "
                "others."
            )
        # Only add this once
        new_doc.append({"local": duplicate_key, "title": titles[0]})

    # Add none duplicate-keys
    new_doc.extend([doc for doc in model_doc if counts[doc["local"]] == 1])

    # Sort


SUMMARY:
**Concise Summary**

The `clean_model_doc_toc` function cleans the table of content in model documentation by removing duplicates and sorting models alphabetically.

**Main Purpose and Functionality**

* Cleans a list of dictionaries (`model_doc`) extracted from `_toctree.yml` by:
	+ Counting occurrences of each local key
	+ Identifying duplicate keys with multiple titles
	+ Removing all but one version of each duplicate title
* Returns the cleaned list sorted alphabetically by title.

**Key Implementation Details**

* Uses `defaultdict` to count occurrences of each local key.
* Utilizes a lambda function for sorting by title (case-insensitive).
* Raises a ValueError if multiple versions of a title are found for a duplicate key.

**Notable Dependencies or Requirements**

* Uses Python's built-in `List`, `dict`, and `defaultdict` types.
* Assumes the presence of an `_toctree.yml` file with documentation data.


key: chroma:document
string_value:
CODE:

def check_model_doc(overwrite: bool = False):
    """
    Check that the content of the table of content in `_toctree.yml` is clean (no duplicates and sorted for the model
    API doc) and potentially auto-cleans it.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether to just check if the TOC is clean or to auto-clean it (when `overwrite=True`).
    """
    with open(PATH_TO_TOC, encoding="utf-8") as f:
        content = yaml.safe_load(f.read())

    # Get to the API doc
    api_idx = 0
    while content[api_idx]["title"] != "API":
        api_idx += 1
    api_doc = content[api_idx]["sections"]

    # Then to the model doc
    model_idx = 0
    while api_doc[model_idx]["title"] != "Models":
        model_idx += 1

    model_doc = api_doc[model_idx]["sections"]

    # Extract the modalities and clean them one by one.
    modalities_docs = [(idx, section) for idx, section in enumerate(model_doc) if "sections" in section]
    diff = False
    for idx, modality_doc in modalities_docs:
        old_modality_doc = modality_doc["sections"]
        new_modality_doc = clean_model_doc_toc(old_modality_doc)

        if old_modality_doc != new_modality_doc:
            diff = True
            if overwrite:
                model_doc[idx]["sections"] = new_modality_doc

    if diff:
        if overwrite:
            api_doc[model_idx]["sections"] = model_doc
            content[api_idx]["sections"] = api_doc
            with open(PATH_TO_TOC, "w", encoding="utf-8") as f:
                f.write(yaml.dump(content, allow_unicode=True))
        else:
            raise ValueError(
                "The model doc part of the table of content is not properly sorted, run `make style` to fix this."


SUMMARY:
**Summary: check_model_doc function**

The `check_model_doc` function validates and potentially auto-cleans the table of contents (`_toctree.yml`) for the model API doc. It checks for:

* Duplicates: A duplicate title is found, indicating an invalid TOC structure.
* Sorting: The TOC contains unsorted sections.

**Key Implementation Details**

1. Reads `PATH_TO_TOC` file using `yaml.safe_load()`.
2. Iterates to find the "API" and "Models" section indices in the content tree.
3. Extracts modalities documents from the "Models" section.
4. Cleans each modality document using `clean_model_doc_toc()`.

**Notable Dependencies/Requirements**

* `yaml`: for parsing the `_toctree.yml` file.
* `PATH_TO_TOC`: a constant representing the file path to `._toctree.yml`.

**Interfaces and Method Signatures**

1. `yaml.safe_load()`: loads YAML data from a string.
2. `clean_model_doc_toc()`: cleans a model doc using unknown implementation details.

This function appears to be part of a larger project for generating and maintaining documentation, possibly in a MarkDown or Sphinx-based environment.


key: chroma:document
string_value:
CODE:

def create_script(target_test):
    """Create a python script to be run by `git bisect run` to determine if `target_test` passes or fails.
    If a test is not found in a commit, the script with exit code `0` (i.e. `Success`).

    Args:
        target_test (`str`): The test to check.

    Returns:
        `str`: The script to be run by `git bisect run`.
    """

    script = f"""
import os
import subprocess

result = subprocess.run(
    ["python3", "-m", "pytest", "-v", f"{target_test}"],
    capture_output = True,
    text=True,
)
print(result.stdout)

if len(result.stderr) > 0:
    if "ERROR: file or directory not found: " in result.stderr:
        print("test not found in this commit")
        exit(0)
    else:
        print(f"pytest failed to run: {{result.stderr}}")
        exit(-1)
elif f"{target_test} FAILED" in result.stdout:
    print("test failed")
    exit(2)

exit(0)
"""

    with open("target_script.py", "w") as fp:


SUMMARY:
**Summary:**

The `create_script` function generates Python script to be run by `git bisect run` for verifying the outcome of test `target_test`. The script imports required modules, runs the specified test using `pytest`, and outputs the result.

**Key Implementation Details:**

* Runs `python3 -m pytest -v <test_name>` with specified `target_test`
* Checks if error or failure message is present in standard output/error
* Exits with success (0) or failure (-1/2) depending on validation

**Notable Dependencies/Requirements:**

* `os` and `subprocess` modules for executing external commands
* `pytest` module for running tests


key: chroma:document
string_value:
CODE:

def find_bad_commit(target_test, start_commit, end_commit):
    """Find (backward) the earliest commit between `start_commit` and `end_commit` at which `target_test` fails.

    Args:
        target_test (`str`): The test to check.
        start_commit (`str`): The latest commit.
        end_commit (`str`): The earliest commit.

    Returns:
        `str`: The earliest commit at which `target_test` fails.
    """

    if start_commit == end_commit:
        return start_commit

    create_script(target_test=target_test)

    bash = f"""
git bisect reset
git bisect start {start_commit} {end_commit}
git bisect run python3 target_script.py
"""

    with open("run_git_bisect.sh", "w") as fp:
        fp.write(bash.strip())

    result = subprocess.run(
        ["bash", "run_git_bisect.sh"],
        capture_output=True,
        text=True,
    )
    print(result.stdout)

    if "error: bisect run failed" in result.stderr:
        index = result.stderr.find("error: bisect run failed")
        bash_error = result.stderr[index:]

        error_msg = f"Error when running git bisect:\nbash error: {bash_error}"

        pattern = "pytest failed to run: .+"
        pytest_errors = re.findall(pattern, result.stdout)
        if len(pytest_errors) > 0:
            pytest_error = pytest_errors[0]
            index = pytest_error.find("pytest failed to run: ")
            index += len("pytest failed to run: ")
            pytest_error = pytest_error[index:]
            error_msg += f"pytest error: {pytest_error}"

        raise ValueError(error_msg)

    pattern = r"(.+) is the first bad commit"
    commits = re.findall(pattern, result.stdout)

    bad_commit = None
    if len(commits) > 0:
        bad_commit = commits[0]

    print(f"Between `start_commit` {start_commit} and `end_commit` {end_commit}")
    print(f"bad_commit: {bad_commit}\n")



SUMMARY:
**Function Summary: find_bad_commit**

**Purpose:** Find the earliest commit between two given commits at which a specified test fails.

### Functionality:

1.  **Initial Check**: Verifies if `start_commit` and `end_commit` are equal, returning `start_commit` in this case.
2.  Creates a bash script to reset git bisect mode and start it with the provided `target_test`.
3.  Runs the `run_git_bisect.sh` script using `subprocess.run`, capturing its output and error messages.
4.  Checks for errors, specifically "bisect run failed". If found, extracts the error message and continues.
5.  Seeks the first occurrence of "first bad commit" in the standard output to identify the earliest failing commit.

### Key Implementation Details:

*   `git bisect` mode is managed using bash scripts and subprocesses.
*   Standard output and error messages are analyzed for errors or test failures.
*   Regular expressions (`re`) are used to extract specific information from the output.

**Key Method Signatures:**

*   `create_script(target_test)`
*   `subprocess.run(["bash", "run_git_bisect.sh"])`

**Dependencies and Interfaces:**

*   External dependencies:
    *   `git` command
    *   `python3` for running script
    *   `re` (regular expression library) for pattern matching

**Interfaces:**

| Interface | Description                |
| :-------- | -------------------------- |
| `target_test` | Target test to check          |

### Notable Variability:

*   Initial commit check logic.
*   Bash script and subprocess execution.

This analysis highlights the primary purpose and key implementation details of the provided function, facilitating a deeper understanding of its functionality and potential areas for optimization.


key: chroma:document
string_value:
CODE:

def get_commit_info(commit):
    """Get information for a commit via `api.github.com`."""
    pr_number = None
    author = None
    merged_author = None

    url = f"https://api.github.com/repos/huggingface/transformers/commits/{commit}/pulls"
    pr_info_for_commit = requests.get(url).json()

    if len(pr_info_for_commit) > 0:
        pr_number = pr_info_for_commit[0]["number"]

        url = f"https://api.github.com/repos/huggingface/transformers/pulls/{pr_number}"
        pr_for_commit = requests.get(url).json()
        author = pr_for_commit["user"]["login"]
        merged_author = pr_for_commit["merged_by"]["login"]

    if author is None:
        url = f"https://api.github.com/repos/huggingface/transformers/commits/{commit}"
        commit_info = requests.get(url).json()
        author = commit_info["author"]["login"]



SUMMARY:
**Summary**

The `get_commit_info` function retrieves information about a GitHub commit, including the pull request number, author, and merged author. It makes API requests to `api.github.com` to fetch data.

* Purpose: Fetch commit metadata via GitHub API.
* Implementation Details:
	+ Retrieves PR info for a given commit and extracts the PR number.
	+ Fetches details about the associated PR, including author and merged author, if available.
	+ Fallbacks to fetching original commit information using `commit` URL if no PR author is found.
	+ Returns a dictionary with commit-related data.

**Key Method Signsatures/Interfaces**

* `requests.get()`: Used for API requests (`url = f..."`)

**Notable Dependencies**

* `requests`
* `json` ( implicit, handling JSON responses from GitHub API)
* `user`, `login` (accessed through PR and commit responses)

**Interface Inheritance/Implementation**

None.

This function leverages the GitHub API to collect metadata. The implementation caters to both cases when a PR is associated with the commit and when it's not.


key: chroma:document
string_value:
CODE:

def onnx_compliancy(saved_model_path, strict, opset):
    saved_model = SavedModel()
    onnx_ops = []

    with open(os.path.join(REPO_PATH, "utils", "tf_ops", "onnx.json")) as f:
        onnx_opsets = json.load(f)["opsets"]

    for i in range(1, opset + 1):
        onnx_ops.extend(onnx_opsets[str(i)])

    with open(saved_model_path, "rb") as f:
        saved_model.ParseFromString(f.read())

    model_op_names = set()

    # Iterate over every metagraph in case there is more than one (a saved model can contain multiple graphs)
    for meta_graph in saved_model.meta_graphs:
        # Add operations in the graph definition
        model_op_names.update(node.op for node in meta_graph.graph_def.node)

        # Go through the functions in the graph definition
        for func in meta_graph.graph_def.library.function:
            # Add operations in each function
            model_op_names.update(node.op for node in func.node_def)

    # Convert to list, sorted if you want
    model_op_names = sorted(model_op_names)
    incompatible_ops = []

    for op in model_op_names:
        if op not in onnx_ops and op not in INTERNAL_OPS:
            incompatible_ops.append(op)

    if strict and len(incompatible_ops) > 0:
        raise Exception(f"Found the following incompatible ops for the opset {opset}:\n" + incompatible_ops)
    elif len(incompatible_ops) > 0:
        print(f"Found the following incompatible ops for the opset {opset}:")
        print(*incompatible_ops, sep="\n")
    else:


SUMMARY:
**Function Summary: `onnx_compliancy`**

### Purpose and Functionality

The `onnx_compliancy` function checks if a saved model (in ONNX format) is compliant with the specified ONNX opset version. It parses the saved model, extracts its operations, and verifies that they are compatible with the given opset.

**Key Implementation Details**

* Extracts ONNX opsets from `onnx.json` file
* Parses saved model from file using `SavedModel.ParseFromString`
* Iterates over multiple graphs in saved model (if present)
* Compares extracted operation names from saved model to list of compatible operations (`onnx_ops`) and internal operations (`INTERNAL_OPS`)
* Raises exception if strict mode is enabled and incompatible ops are found

**Notable Dependencies and Requirements**

* `SavedModel` class
* `json` module for parsing `onnx.json` file
* `os` module for getting repository path (`REPO_PATH`)
* External dependencies:
	+ `onnx_ops` (list of compatible ONNX opsets)
	+ `INTERNAL_OPS` (list of internal operations)


key: chroma:document
string_value:
CODE:

class QuantizationMessage(Message):
    def __init__(
        self,
        title: str,
        results: Dict,
    ):
        self.title = title

        # Failures and success of the modeling tests
        self.n_success = sum(r["success"] for r in results.values())
        self.single_gpu_failures = sum(r["failed"]["single"] for r in results.values())
        self.multi_gpu_failures = sum(r["failed"]["multi"] for r in results.values())
        self.n_failures = self.single_gpu_failures + self.multi_gpu_failures

        self.n_tests = self.n_failures + self.n_success
        self.results = results
        self.thread_ts = None

    @property
    def payload(self) -> str:
        blocks = [self.header]

        if self.n_failures > 0:
            blocks.append(self.failures_overwiew)
            blocks.append(self.failures_detailed)

        if self.n_failures == 0:
            blocks.append(self.no_failures)

        return json.dumps(blocks)

    @property
    def time(self) -> str:
        all_results = self.results.values()
        time_spent = []
        for r in all_results:
            if len(r["time_spent"]):
                time_spent.extend([x for x in r["time_spent"].split(", ") if len(x.strip())])
        total_secs = 0

        for time in time_spent:
            time_parts = time.split(":")

            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.
            if len(time_parts) == 1:
                time_parts = [0, 0, time_parts[0]]

            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])
            total_secs += hours * 3600 + minutes * 60 + seconds

        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60
        return f"{int(hours)}h{int(minutes)}m{int(seconds)}s"

    @property
    def failures_overwiew(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": (
                    f"There were {self.n_failures} failures, out of {self.n_tests} tests.\n"
                    f"The suite ran in {self.time}."
                ),
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def failures_detailed(self) -> Dict:
        failures = {k: v["failed"] for k, v in self.results.items()}

        individual_reports = []
        for key, value in failures.items():
            device_report = self.get_device_report(value)
            if sum(value.values()):
                report = f"{device_report}{key}"
                individual_reports.append(report)

        header = "Single |  Multi | Category\n"
        failures_report = prepare_reports(
            title="The following quantization tests had failures", header=header, reports=individual_reports
        )

        return {"type": "section", "text": {"type": "mrkdwn", "text": failures_report}}

    def post(self):
        payload = self.payload
        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(payload)}))

        text = f"{self.n_failures} failures out of {self.n_tests} tests," if self.n_failures else "All tests passed."

        self.thread_ts = client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            blocks=payload,
            text=text,
        )

    def post_reply(self):
        if self.thread_ts is None:
            raise ValueError("Can only post reply if a post has been made.")

        for job, job_result in self.results.items():
            if len(job_result["failures"]):
                for device, failures in job_result["failures"].items():
                    blocks = self.get_reply_blocks(
                        job,
                        job_result,
                        failures,
                        device,
                        text=f'Number of failures: {job_result["failed"][device]}',
                    )

                    print("Sending the following reply")
                    print(json.dumps({"blocks": blocks}))

                    client.chat_postMessage(
                        channel="#transformers-ci-daily-quantization",
                        text=f"Results for {job}",
                        blocks=blocks,
                        thread_ts=self.thread_ts["ts"],
                    )


SUMMARY:
**Summary: QuantizationMessage Class**

This class, `QuantizationMessage`, inherits from the `Message` class and represents a message containing quantization testing results.

**Main Purpose and Functionality**
The main purpose of this class is to store and format the results of quantization tests in a JSON payload that can be sent through Slack.

**Implementation Details**

*   **Initialization**: The constructor takes two parameters: `title` (a string) and `results` (a dictionary).
    *   `results`: A dictionary containing test results, including failure information.
        *   `n_success`
        *   `single_gpu_failures`
        *   `multi_gpu_failures`
        *   `n_failures`
        *   `n_tests`
        *   `thread_ts` (initially None)
*   **Properties**:
    *   `payload`: Returns the formatted JSON payload as a string, including optional sections for failure overview and detailed reports.
        *   `blocks`: A list of JSON blocks containing the message content.
        *   Includes failuer section
    *   `time`: Calculates and returns the total time spent on all tests in hours, minutes, and seconds.
*   **Methods**:
    *   `post()`: Sends the message payload to a Slack channel using the client's chat post message method.
    *   `post_reply()`: Posts replies for individual jobs with failures.


key: chroma:document
string_value:
CODE:

def get_new_python_files_between_commits(base_commit: str, commits: List[str]) -> List[str]:
    """
    Get the list of added python files between a base commit and one or several commits.

    Args:
        repo (`git.Repo`):
            A git repository (for instance the Transformers repo).
        base_commit (`str`):
            The commit reference of where to compare for the diff. This is the current commit, not the branching point!
        commits (`List[str]`):
            The list of commits with which to compare the repo at `base_commit` (so the branching point).

    Returns:
        `List[str]`: The list of python files added between a base commit and one or several commits.
    """
    code_diff = []
    for commit in commits:
        for diff_obj in commit.diff(base_commit):
            # We always add new python files
            if diff_obj.change_type == "A" and diff_obj.b_path.endswith(".py"):
                code_diff.append(diff_obj.b_path)



SUMMARY:
**Function Summary:**

`get_new_python_files_between_commits`

Returns a list of added Python files between a base commit (`base_commit`) and one or several commits (`commits`). Compares each commit's diff against the base commit to identify new Python files.

**Key Implementation Details:**

* Uses `git.Repo.diff()` to compare each commit's diff against the base commit.
* Iterates through a list of commits (`commits`) using the `diff()` method.
* Identifies new Python files added in each commit by checking for file creation type ("A") and file extension (`.py`).
* Returns a list of paths to newly created Python files.

**Notable Dependencies/Requirements:**

* `git.Repo`: Git repository access
* `List[str]`: List of commit references

**Primary Purpose:**
Identify new Python files added between a base commit and one or several commits in a Git repository.


key: chroma:document
string_value:
CODE:

def get_new_python_files() -> List[str]:
    """
    Return a list of python files that have been added between the current head and the main branch.

    Returns:
        `List[str]`: The list of python files added.
    """
    repo = Repo(PATH_TO_REPO)

    try:
        # For the cases where the main branch exists locally
        main = repo.refs.main
    except AttributeError:
        # On GitHub Actions runners, it doesn't have local main branch
        main = repo.remotes.origin.refs.main

    print(f"main is at {main.commit}")
    print(f"Current head is at {repo.head.commit}")

    branching_commits = repo.merge_base(main, repo.head)
    for commit in branching_commits:
        print(f"Branching commit: {commit}")


SUMMARY:
**Function Summary**

`get_new_python_files()` returns a list of Python files added between the current Git head and the `main` branch.

**Key Implementation Details**

* Uses `Repo` class to interact with the GitHub repository
* Checks if `main` branch exists locally, falling back to remote `origin` in GitHub Actions runners
* Retrieves merge base commits using `repo.merge_base`
* Returns list of new Python files added using `get_new_python_files_between_commits`

**Notable Dependencies and Requirements**

* `Repo` class from `pathlib` library ( implied but not explicitly stated)
* External dependency on GitHub repository at `PATH_TO_REPO`


key: chroma:document
string_value:
CODE:

def get_new_model():
    new_files = get_new_python_files()
    reg = re.compile(r"src/transformers/models/(.*)/modeling_.*\.py")

    new_model = ""
    for x in new_files:
        find_new_model = reg.findall(x)
        if len(find_new_model) > 0:
            new_model = find_new_model[0]
            # It's unlikely we have 2 new modeling files in a pull request.
            break


SUMMARY:
**Function Summary: `get_new_model`**

Retrieves the names of newly created Python model files in a Git repository.

**Primary Functionality:**
Parses file contents to extract and return the name of the new model file, assuming only one new modeling file exists in each pull request.

**Key Implementation Details:**

* Uses regular expression `reg` to extract pattern "src/transformers/models/(.*)/modeling_.*\.py"
* Iterates over list of new Python files (`new_files`) and uses `re.findall` to find the matching model file name
* Returns the extracted model file name as a string

**Notable Dependencies or Requirements:**

* `re` (regular expression) module for pattern matching
* `get_new_python_files()` external function or call, returning list of new Python files


key: chroma:document
string_value:
CODE:

def parse_message(message: str) -> str:
    """
    Parses a GitHub pull request's comment to find the models specified in it to run slow CI.

    Args:
        message (`str`): The body of a GitHub pull request's comment.

    Returns:
        `str`: The substring in `message` after `run-slow`, run_slow` or run slow`. If no such prefix is found, the
        empty string is returned.
    """
    if message is None:
        return ""

    message = message.strip().lower()

    # run-slow: model_1, model_2
    if not message.startswith(("run-slow", "run_slow", "run slow")):
        return ""
    message = message[len("run slow") :]
    # remove leading `:`
    while message.strip().startswith(":"):
        message = message.strip()[1:]



SUMMARY:
**Summary**

### Functionality

The `parse_message` function parses a GitHub pull request comment to extract models specified in the message to run slow CI.

### Implementation Details

* **Key Algorithm:** Trims and normalizes user input message.
* **Method Signature:**
	+ `parse_message(message: str) -> str`
* **Design Choice:** Uses multiple prefixes ("run-slow", "run_slow", "run slow") to account for varying notation in comments.

### Dependencies and Requirements

* None explicitly listed, but assumes `str` type for input message.


key: chroma:document
string_value:
CODE:

def get_models(message: str):
    models = parse_message(message)


SUMMARY:
**Summary**

### Function Overview

`get_models(message: str)`

Extracts and preprocesses a list of models from a given message.

### Primary Purpose

Resets and splits the `models` string to separate individual model names, replacing commas with spaces.

### Key Implementation Details

* Uses `parse_message(message)` (exact class/function not provided)
* Applies whitespace manipulation to output string
* Utilizes string splitting (`split()`) on `models`

### Notable Dependencies or Requirements

* `parse_message(message)` (related function not provided)
 
### External Dependencies

List of external dependencies is not specified in the given code chunk


key: chroma:document
string_value:
CODE:

def check_model_names(model_name: str):
    allowed = string.ascii_letters + string.digits + "_"


SUMMARY:
**Summary**

Checks if a `model_name` is valid. A valid model name consists of letters, digits, and underscores only.

* **Primary Purpose**: Validate model names to ensure they comply with the specified format.
* **Key Implementation Details**:
	+ Uses regular expression for string validation (`string.ascii_letters + string.digits + "_"`).
	+ Checks for leading/trailing underscore prefixes using `startswith()` and `endswith()`.
* **Notable Dependencies**:
	+ `string` module for character classification.
* **Interfaces/Inherited Class**: None.

In RAG system notation: 
### model_name_validation
- Description: Validates a model name to ensure it contains only letters, digits, or underscores.
- Primary Purpose: Ensure model names meet the specified format requirement.


key: chroma:document
string_value:
CODE:

def process_file(modular_file_path, generated_modeling_content, file_type="modeling_", fix_and_overwrite=False):
    file_name_prefix = file_type.split("*")[0]
    file_name_suffix = file_type.split("*")[-1] if "*" in file_type else ""
    file_path = modular_file_path.replace("modular_", f"{file_name_prefix}_").replace(".py", f"{file_name_suffix}.py")
    # Read the actual modeling file
    with open(file_path, "r") as modeling_file:
        content = modeling_file.read()
    output_buffer = StringIO(generated_modeling_content[file_type][0])
    output_buffer.seek(0)
    output_content = output_buffer.read()
    diff = difflib.unified_diff(
        output_content.splitlines(),
        content.splitlines(),
        fromfile=f"{file_path}_generated",
        tofile=f"{file_path}",
        lineterm="",
    )
    diff_list = list(diff)
    # Check for differences
    if diff_list:
        if fix_and_overwrite:
            with open(file_path, "w") as modeling_file:
                modeling_file.write(generated_modeling_content[file_type][0])
            console.print(f"[bold blue]Overwritten {file_path} with the generated content.[/bold blue]")
        else:
            console.print(f"\n[bold red]Differences found between the generated code and {file_path}:[/bold red]\n")
            diff_text = "\n".join(diff_list)
            syntax = Syntax(diff_text, "diff", theme="ansi_dark", line_numbers=True)
            console.print(syntax)
        return 1
    else:
        console.print(f"[bold green]No differences found for {file_path}.[/bold green]")


SUMMARY:
**Function Summary**

`process_file(modular_file_path, generated_modeling_content, file_type="modeling_", fix_and_overwrite=False)`

**Purpose and Functionality**
Process and compare the content of two modeling files located in `modular_file_path`.

**Main Algorithms/P Patterns/Design Choices**

* Reading and comparing file contents using `difflib.unified_diff` to identify differences.
* Overwriting or printing differences based on `fix_and_overwrite` flag.

**Key Method Signatures and Interfaces**

- `difflib.unified_diff(fromfile, tofile, lineterm="")`
- `console.print()`

**Notable Dependencies and Requirements**

- `StringIO`
- `difflib`
- `console` (assuming a custom console object or module)
- `generated_modeling_content[file_type]` referencing an external data structure.


key: chroma:document
string_value:
CODE:

def compare_files(modular_file_path, fix_and_overwrite=False):
    # Generate the expected modeling content
    generated_modeling_content = convert_modular_file(modular_file_path)
    diff = 0
    for file_type in generated_modeling_content.keys():
        diff += process_file(modular_file_path, generated_modeling_content, file_type, fix_and_overwrite)


SUMMARY:
**Summary:**

`compare_files` function compares the expected modeling content with actual files specified by `modular_file_path`. It iterates through each file type, processing them using `process_file`, and returns the cumulative difference (`diff`). 

**Key Implementation Details:**

* Generates expected modeling content using `convert_modular_file(modular_file_path)`
	+ Input parameter: `modular_file_path`
	+ Output parameter: `generated_modeling_content` dictionary containing different file types
* Iterates through each file type in `generated_modeling_content` and processes them using `process_file(modular_file_path, generated_modeling_content, file_type, fix_and_overwrite)`
	+ Input parameters: `modular_file_path`, `generated_modeling_content`, `file_type`
	+ Output parameter: cumulative difference (`diff`)
* Returns the cumulative difference (`diff`) after processing all files

**Notable Dependencies/Requirements:**

* `convert_modular_file` function
* `process_file` function with parameters `modular_file_path`, `generated_modeling_content`, `file_type`, and `fix_and_overwrite`
 
This summary aims to preserve the original method's flow, focus on key implementation details, and reference related functions by exact name.


key: chroma:document
string_value:
CODE:

def find_backend(line: str) -> Optional[str]:
    """
    Find one (or multiple) backend in a code line of the init.

    Args:
        line (`str`): A code line of the main init.

    Returns:
        Optional[`str`]: If one (or several) backend is found, returns it. In the case of multiple backends (the line
        contains `if is_xxx_available() and `is_yyy_available()`) returns all backends joined on `_and_` (so
        `xxx_and_yyy` for instance).
    """
    if _re_test_backend.search(line) is None:
        return None
    backends = [b[0] for b in _re_backend.findall(line)]
    backends.sort()


SUMMARY:
**Summary**

`find_backend` function identifies and extracts multiple backend names (`xxx_and_yyy`) from a given code line of the main `init` file. It performs a case-insensitive search using regular expression `_re_test_backend.search(line)`.

**Implementation Details**

* Uses two regular expressions: `_re_test_backend` to search for the backend pattern and `_re_backend` to extract the backend names.
* Extracts backend names from the line using a list comprehension (`[b[0] for b in _re_backend.findall(line)]`) and sorts them alphabetically.

**Return Value**

Returns all extracted backend names joined on `_and_`, or `None` if no backend is found (`if _re_test_backend.search(line) is None`).


key: chroma:document
string_value:
CODE:

def parse_init(init_file) -> Optional[Tuple[Dict[str, List[str]], Dict[str, List[str]]]]:
    """
    Read an init_file and parse (per backend) the `_import_structure` objects defined and the `TYPE_CHECKING` objects
    defined.

    Args:
        init_file (`str`): Path to the init file to inspect.

    Returns:
        `Optional[Tuple[Dict[str, List[str]], Dict[str, List[str]]]]`: A tuple of two dictionaries mapping backends to list of
        imported objects, one for the `_import_structure` part of the init and one for the `TYPE_CHECKING` part of the
        init. Returns `None` if the init is not a custom init.
    """
    with open(init_file, "r", encoding="utf-8", newline="\n") as f:
        lines = f.readlines()

    # Get the to `_import_structure` definition.
    line_index = 0
    while line_index < len(lines) and not lines[line_index].startswith("_import_structure = {"):
        line_index += 1

    # If this is a traditional init, just return.
    if line_index >= len(lines):
        return None

    # First grab the objects without a specific backend in _import_structure
    objects = []
    while not lines[line_index].startswith("if TYPE_CHECKING") and find_backend(lines[line_index]) is None:
        line = lines[line_index]
        # If we have everything on a single line, let's deal with it.
        if _re_one_line_import_struct.search(line):
            content = _re_one_line_import_struct.search(line).groups()[0]
            imports = re.findall(r"\[([^\]]+)\]", content)
            for imp in imports:
                objects.extend([obj[1:-1] for obj in imp.split(", ")])
            line_index += 1
            continue
        single_line_import_search = _re_import_struct_key_value.search(line)
        if single_line_import_search is not None:
            imports = [obj[1:-1] for obj in single_line_import_search.groups()[0].split(", ") if len(obj) > 0]
            objects.extend(imports)
        elif line.startswith(" " * 8 + '"'):
            objects.append(line[9:-3])
        line_index += 1

    # Those are stored with the key "none".
    import_dict_objects = {"none": objects}

    # Let's continue with backend-specific objects in _import_structure
    while not lines[line_index].startswith("if TYPE_CHECKING"):
        # If the line is an if not is_backend_available, we grab all objects associated.
        backend = find_backend(lines[line_index])
        # Check if the backend declaration is inside a try block:
        if _re_try.search(lines[line_index - 1]) is None:
            backend = None

        if backend is not None:
            line_index += 1

            # Scroll until we hit the else block of try-except-else
            while _re_else.search(lines[line_index]) is None:
                line_index += 1

            line_index += 1

            objects = []
            # Until we unindent, add backend objects to the list
            while len(lines[line_index]) <= 1 or lines[line_index].startswith(" " * 4):
                line = lines[line_index]
                if _re_import_struct_add_one.search(line) is not None:
                    objects.append(_re_import_struct_add_one.search(line).groups()[0])
                elif _re_import_struct_add_many.search(line) is not None:
                    imports = _re_import_struct_add_many.search(line).groups()[0].split(", ")
                    imports = [obj[1:-1] for obj in imports if len(obj) > 0]
                    objects.extend(imports)
                elif _re_between_brackets.search(line) is not None:
                    imports = _re_between_brackets.search(line).groups()[0].split(", ")
                    imports = [obj[1:-1] for obj in imports if len(obj) > 0]
                    objects.extend(imports)
                elif _re_quote_object.search(line) is not None:
                    objects.append(_re_quote_object.search(line).groups()[0])
                elif line.startswith(" " * 8 + '"'):
                    objects.append(line[9:-3])
                elif line.startswith(" " * 12 + '"'):
                    objects.append(line[13:-3])
                line_index += 1

            import_dict_objects[backend] = objects
        else:
            line_index += 1

    # At this stage we are in the TYPE_CHECKING part, first grab the objects without a specific backend
    objects = []
    while (
        line_index < len(lines)
        and find_backend(lines[line_index]) is None
        and not lines[line_index].startswith("else")
    ):
        line = lines[line_index]
        single_line_import_search = _re_import.search(line)
        if single_line_import_search is not None:
            objects.extend(single_line_import_search.groups()[0].split(", "))
        elif line.startswith(" " * 8):
            objects.append(line[8:-2])
        line_index += 1

    type_hint_objects = {"none": objects}

    # Let's continue with backend-specific objects
    while line_index < len(lines):
        # If the line is an if is_backend_available, we grab all objects associated.
        backend = find_backend(lines[line_index])
        # Check if the backend declaration is inside a try block:
        if _re_try.search(lines[line_index - 1]) is None:
            backend = None

        if backend is not None:
            line_index += 1

            # Scroll until we hit the else block of try-except-else
            while _re_else.search(lines[line_index]) is None:
                line_index += 1

            line_index += 1

            objects = []
            # Until we unindent, add backend objects to the list
            while len(lines[line_index]) <= 1 or lines[line_index].startswith(" " * 8):
                line = lines[line_index]
                single_line_import_search = _re_import.search(line)
                if single_line_import_search is not None:
                    objects.extend(single_line_import_search.groups()[0].split(", "))
                elif line.startswith(" " * 12):
                    objects.append(line[12:-2])
                line_index += 1

            type_hint_objects[backend] = objects
        else:
            line_index += 1



SUMMARY:
**Function Summary: `parse_init`**

Parsing an init file to extract import and TYPE_CHECKING definitions.

**Purpose:** Read an init file, identifying and parsing the `_import_structure` and `TYPE_CHECKING` objects defined in it.

**Primary Purpose:** The function reads a `.init` file and returns two dictionaries mapping backends to lists of imported objects: one for the `_import_structure` section and one for the `TYPE_CHECKING` section.

**Key Implementation Details:**

*   Checks for `_import_structure = { ... }` definitions to parse backend-specific imports.
*   Iterates through lines to identify and extract:
    *   Import structures without a specific backend (`_re_one_line_import_struct.search`)
    *   Backend-specific imports within the try-except-else block (`_re_backend_find`)
    *   TYPE_CHECKING objects without a specific backend (`_re_import.search`)
*   Parses import definitions using regular expressions (e.g., `_import_struct_add_many`, `_re_between_brackets`)

**Notable Dependencies and Requirements:**

*   The function uses regular expression classes (`_re_one_line_import_struct`, `_re_backend_find`, etc.) to parse the init file.
*   It relies on `find_backend()` function, but its implementation is not provided in this code snippet.

**Return Value:** A tuple of two dictionaries:
    *   `import_dict_objects`: Mapping backends to lists of import structures without a specific backend (`"none"`).
    *   `type_hint_objects`: Mapping backends (if applicable) to lists of TYPE_CHECKING objects without a specific backend (`"none"`).


key: chroma:document
string_value:
CODE:

def analyze_results(import_dict_objects: Dict[str, List[str]], type_hint_objects: Dict[str, List[str]]) -> List[str]:
    """
    Analyze the differences between _import_structure objects and TYPE_CHECKING objects found in an init.

    Args:
        import_dict_objects (`Dict[str, List[str]]`):
            A dictionary mapping backend names (`"none"` for the objects independent of any specific backend) to
            list of imported objects.
        type_hint_objects (`Dict[str, List[str]]`):
            A dictionary mapping backend names (`"none"` for the objects independent of any specific backend) to
            list of imported objects.

    Returns:
        `List[str]`: The list of errors corresponding to mismatches.
    """

    def find_duplicates(seq):
        return [k for k, v in collections.Counter(seq).items() if v > 1]

    # If one backend is missing from the other part of the init, error early.
    if list(import_dict_objects.keys()) != list(type_hint_objects.keys()):
        return ["Both sides of the init do not have the same backends!"]

    errors = []
    # Find all errors.
    for key in import_dict_objects.keys():
        # Duplicate imports in any half.
        duplicate_imports = find_duplicates(import_dict_objects[key])
        if duplicate_imports:
            errors.append(f"Duplicate _import_structure definitions for: {duplicate_imports}")
        duplicate_type_hints = find_duplicates(type_hint_objects[key])
        if duplicate_type_hints:
            errors.append(f"Duplicate TYPE_CHECKING objects for: {duplicate_type_hints}")

        # Missing imports in either part of the init.
        if sorted(set(import_dict_objects[key])) != sorted(set(type_hint_objects[key])):
            name = "base imports" if key == "none" else f"{key} backend"
            errors.append(f"Differences for {name}:")
            for a in type_hint_objects[key]:
                if a not in import_dict_objects[key]:
                    errors.append(f"  {a} in TYPE_HINT but not in _import_structure.")
            for a in import_dict_objects[key]:
                if a not in type_hint_objects[key]:
                    errors.append(f"  {a} in _import_structure but not in TYPE_HINT.")


SUMMARY:
**Function Summary**

`analyze_results` analyzes the discrepancies between `_import_structure` objects and `TYPE_CHECKING` objects found in an init.

**Main Purpose**
Identifies duplicate imports, missing imports, and incorrect backend names across both structures.

**Key Implementation Details**

* Verifies that both input dictionaries have identical keys (backends).
* Uses a custom function `find_duplicates` to detect duplicate values in lists.
* Iterates through each key-value pair, comparing the two dictionaries for:

    * Duplicate imports
    * Missing imports

**Notable Dependencies/Requirements**

* Python built-in libraries: `collections`
* External dependencies:
	+ None (self-contained)
* Implemented interfaces:
	+ No explicit interface implementation; relies on dictionary comparisons.


key: chroma:document
string_value:
CODE:

def check_all_inits():
    """
    Check all inits in the transformers repo and raise an error if at least one does not define the same objects in
    both halves.
    """
    failures = []
    for root, _, files in os.walk(PATH_TO_TRANSFORMERS):
        if "__init__.py" in files:
            fname = os.path.join(root, "__init__.py")
            objects = parse_init(fname)
            if objects is not None:
                errors = analyze_results(*objects)
                if len(errors) > 0:
                    errors[0] = f"Problem in {fname}, both halves do not define the same objects.\n{errors[0]}"
                    failures.append("\n".join(errors))
    if len(failures) > 0:


SUMMARY:
**Function Summary:**

`check_all_inits()`

Checks all initalization files (`__init__.py`) in the transformers repository for potential object definition discrepancies between both halves. Raises an error if any file fails to define the same objects.

**Key Implementation Details:**

* Iterates through `__init__.py` files in the transformers repository using `os.walk`.
* Parses initialization file contents using `parse_init(fname)`.
* Analyzes results using `analyze_results(*objects)` and accumulates errors.
* Raises a `ValueError` with error messages if any discrepancies are found.

**Notable Dependencies:**

* `os`, `PATH_TO_TRANSFORMERS`
* `parse_init`, `analyze_results` (external functions not shown in this code snippet)


key: chroma:document
string_value:
CODE:

def get_transformers_submodules() -> List[str]:
    """
    Returns the list of Transformers submodules.
    """
    submodules = []
    for path, directories, files in os.walk(PATH_TO_TRANSFORMERS):
        for folder in directories:
            # Ignore private modules
            if folder.startswith("_"):
                directories.remove(folder)
                continue
            # Ignore leftovers from branches (empty folders apart from pycache)
            if len(list((Path(path) / folder).glob("*.py"))) == 0:
                continue
            short_path = str((Path(path) / folder).relative_to(PATH_TO_TRANSFORMERS))
            submodule = short_path.replace(os.path.sep, ".")
            submodules.append(submodule)
        for fname in files:
            if fname == "__init__.py":
                continue
            short_path = str((Path(path) / fname).relative_to(PATH_TO_TRANSFORMERS))
            submodule = short_path.replace(".py", "").replace(os.path.sep, ".")
            if len(submodule.split(".")) == 1:
                submodules.append(submodule)


SUMMARY:
**Summary**

`get_transformers_submodules()` function returns a list of Transformers submodules by traversing the directory tree rooted at `PATH_TO_TRANSFORMERS`. It filters out private modules, empty folders, and initializes that are not submodule names.

**Key Implementation Details**

* Uses `os.walk()` to traverse directory tree
* Checks for file patterns in folders (`*.py`, `__init__.py`) and files (excluding `__init__.py`)
* Normalizes folder and file paths to standardize submodule names

**Notable Dependencies or Requirements**

* `PATH_TO_TRANSFORMERS`
* `os` module for walking directories
* `Path` class from `pathlib` library for path manipulation


key: chroma:document
string_value:
CODE:

def check_submodules():
    """
    Check all submodules of Transformers are properly registered in the main init. Error otherwise.
    """
    # This is to make sure the transformers module imported is the one in the repo.
    from transformers.utils import direct_transformers_import

    transformers = direct_transformers_import(PATH_TO_TRANSFORMERS)

    import_structure_keys = set(transformers._import_structure.keys())
    # This contains all the base keys of the _import_structure object defined in the init, but if the user is missing
    # some optional dependencies, they may not have all of them. Thus we read the init to read all additions and
    # (potentiall re-) add them.
    with open(os.path.join(PATH_TO_TRANSFORMERS, "__init__.py"), "r") as f:
        init_content = f.read()
    import_structure_keys.update(set(re.findall(r"import_structure\[\"([^\"]*)\"\]", init_content)))

    module_not_registered = [
        module
        for module in get_transformers_submodules()
        if module not in IGNORE_SUBMODULES and module not in import_structure_keys
    ]

    if len(module_not_registered) > 0:
        list_of_modules = "\n".join(f"- {module}" for module in module_not_registered)
        raise ValueError(
            "The following submodules are not properly registed in the main init of Transformers:\n"
            f"{list_of_modules}\n"
            "Make sure they appear somewhere in the keys of `_import_structure` with an empty list as value."


SUMMARY:
**Summary: `check_submodules()` Function**

* **Purpose:** Validate that all submodules in `Transformers` are properly initialized with their respective import structures.
* **Key Implementation Details:**
	+ Verifies `transformers._import_structure` keys against the expected structure and user-defined imports from `_init.py`.
	+ Compares submodule lists against imported `import_structure_keys`.
	+ Raises a `ValueError` if any submodules are not properly registered.
* **Notable Dependencies & Requirements:**
	+ `direct_transformers_import`: imports transformers module ensuring it's the one in the repo.
	+ `get_transformers_submodules`: recursively finds all submodules of Transformers.
	+ `_init.py`: provides user-defined import structure additions for optional dependencies.


key: chroma:document
string_value:
CODE:

def get_daily_ci_runs(token, num_runs=7):
    """Get the workflow runs of the scheduled (daily) CI.

    This only selects the runs triggered by the `schedule` event on the `main` branch.
    """
    headers = None
    if token is not None:
        headers = {"Accept": "application/vnd.github+json", "Authorization": f"Bearer {token}"}

    # The id of a workflow (not of a workflow run).
    # From a given workflow run (where we have workflow run id), we can get the workflow id by going to
    # https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}
    # and check the `workflow_id` key.
    workflow_id = "90575235"

    url = f"https://api.github.com/repos/huggingface/transformers/actions/workflows/{workflow_id}/runs"
    # On `main` branch + event being `schedule` + not returning PRs + only `num_runs` results
    url += f"?branch=main&event=schedule&exclude_pull_requests=true&per_page={num_runs}"

    result = requests.get(url, headers=headers).json()



SUMMARY:
**Function Summary: `get_daily_ci_runs`**

**Purpose and Functionality:** Retrieves the workflow run history of scheduled CI for the GitHub repository "huggingface/transformers" on the `main` branch filtered by schedule events.

**Key Implementation Details:**

* Employs the API endpoint `https://api.github.com/repos/huggingface/transformers/actions/workflows/{workflow_id}/runs` with query parameters:
	+ Branch: `main`
	+ Event: `schedule`
	+ Exclude pull requests: `true`
	+ Page size: `num_runs`
* Utilizes the `requests.get()` method to send a GET request to the API endpoint, authenticated via Bearer token.

**Notable Dependencies and Requirements:**

* `token`: GitHub API authentication token
* `requests` library for API calls
* External dependencies:
	+ github.com ( GitHub repository)
	+ transformrsactions API


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_runs(token):
    """Get the last completed workflow run id of the scheduled (daily) CI."""
    workflow_runs = get_daily_ci_runs(token)
    workflow_run_id = None
    for workflow_run in workflow_runs:
        if workflow_run["status"] == "completed":
            workflow_run_id = workflow_run["id"]
            break



SUMMARY:
**Summary:**

```markdown
### Function:
#### `get_last_daily_ci_runs(token)`

#### Purpose:
 Retrieves the ID of the most recent completed daily Continuous Integration (CI) workflow.

#### Implementation Details:

* Calls `get_daily_ci_runs(token)` to retrieve daily CI runs
* Iterates through runs, returning the ID of the first complete run (`if "status" in workflow_run:`)
* Returns the most recent completed daily CI run's ID

#### Notable Dependencies:
None

#### Interfaces:
- `get_daily_ci_runs(token)`
```

**RAG Code Tag:**
`func: get_last_daily_ci_runs(token)`:


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_run_commit(token):
    """Get the commit sha of the last completed scheduled daily CI workflow run."""
    workflow_runs = get_daily_ci_runs(token)
    head_sha = None
    for workflow_run in workflow_runs:
        if workflow_run["status"] == "completed":
            head_sha = workflow_run["head_sha"]
            break



SUMMARY:
**Summary**

The `get_last_daily_ci_run_commit` function retrieves the commit SHA of the last completed scheduled daily CI workflow run.

**Key Points:**

* **Purpose:** Returns the SHA of the latest completed CI run.
* **Functionality:** Iterates through completed daily CI runs to find the head SHA associated with the completion.
* **Implementation Details:**
	+ Calls `get_daily_ci_runs` function to retrieve successful workflow runs.
	+ Loops through runs, identifying completed ones and saving their corresponding head SHA.
	+ Returns the head SHA of the last completed run.

**Notable Dependencies/Requirements:**

* `get_daily_ci_runs(token)` - Function called to retrieve daily CI runs.


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_artifacts(artifact_names, output_dir, token):
    """Get the artifacts of last completed workflow run id of the scheduled (daily) CI."""
    workflow_run_id = get_last_daily_ci_runs(token)
    if workflow_run_id is not None:
        artifacts_links = get_artifacts_links(worflow_run_id=workflow_run_id, token=token)
        for artifact_name in artifact_names:
            if artifact_name in artifacts_links:
                artifact_url = artifacts_links[artifact_name]
                download_artifact(
                    artifact_name=artifact_name, artifact_url=artifact_url, output_dir=output_dir, token=token


SUMMARY:
**Function Summary:**

`get_last_daily_ci_artifacts`

 Retrieves and downloads artifacts from the last completed daily CI workflow run.

**Key Implementation Details:**

* Fetches `workflow_run_id` using `get_last_daily_ci_runs`.
* Retrieves artifact links for the specified `workflow_run_id` using `get_artifacts_links`.
* Iterates over `artifact_names`, downloading artifacts matching those names using `download_artifact`.

**Notable Dependencies or Requirements:**

* Utilizes `token` in multiple calls.
* Calls `get_artifacts_links` with `workflow_run_id` and `token`.
* Expects implementation of `download_artifact` function separate from this snippet.

This summary prescribes the main purpose, key algorithms/patterns, and dependencies used within `get_last_daily_ci_artifacts`.


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_reports(artifact_names, output_dir, token):
    """Get the artifacts' content of the last completed workflow run id of the scheduled (daily) CI."""
    get_last_daily_ci_artifacts(artifact_names, output_dir, token)

    results = {}
    for artifact_name in artifact_names:
        artifact_zip_path = os.path.join(output_dir, f"{artifact_name}.zip")
        if os.path.isfile(artifact_zip_path):
            results[artifact_name] = {}
            with zipfile.ZipFile(artifact_zip_path) as z:
                for filename in z.namelist():
                    if not os.path.isdir(filename):
                        # read the file
                        with z.open(filename) as f:
                            results[artifact_name][filename] = f.read().decode("UTF-8")



SUMMARY:
**Summary**

The `get_last_daily_ci_reports` function retrieves the artifacts' content from the last completed daily CI workflow run. It takes three parameters: `artifact_names`, `output_dir`, and `token`.

**Purpose and Functionality**

Retrieves artifacts from the last daily CI workflow run, saving their contents in a dictionary for each artifact.

**Key Implementation Details**

* Iterates over `.zip` files with artifact names matching in the provided directories.
* Reads and decodes file contents using UTF-8 encoding.
* Returns a nested dictionary containing artifact names as keys and their corresponding file contents as values.

**Notable Dependencies or Requirements**

* `os.path.isfile`
* `os.open`
* `zipped zipfile.ZipFile`

Note: This summary assumes the absence of any external dependencies not mentioned in the provided code. If such dependencies exist, please provide them for a more comprehensive analysis.


key: chroma:document
string_value:
CODE:

def extract_time_from_single_job(job):
    """Extract time info from a single job in a GitHub Actions workflow run"""

    job_info = {}

    start = job["started_at"]
    end = job["completed_at"]

    start_datetime = date_parser.parse(start)
    end_datetime = date_parser.parse(end)

    duration_in_min = round((end_datetime - start_datetime).total_seconds() / 60.0)

    job_info["started_at"] = start
    job_info["completed_at"] = end
    job_info["duration"] = duration_in_min



SUMMARY:
**Summary**

This function, `extract_time_from_single_job`, extracts time-related information from a single job in a GitHub Actions workflow run. It:

* Retrieves the start and end timestamps of the job
* Calculates the duration in minutes between these two timestamps
* Stores this duration along with the original timestamps in a dictionary, `job_info`
* Returns the populated `job_info` dictionary

**Key Implementation Details**

* Retrieves dates using date parsing (`date_parser.parse`)
* Calculates duration in minutes from the timestamp difference ( converted to seconds)

**Notable Dependencies or Requirements**

* `date_parser`: assumed dependency for parsing timestamps into datetimes
* Expected input: `job` dictionary with keys "started_at" and "completed_at"
* Returns a dictionary (`job_info`) with keys "started_at", "completed_at", and "duration".


key: chroma:document
string_value:
CODE:

def get_job_time(workflow_run_id, token=None):
    """Extract time info for all jobs in a GitHub Actions workflow run"""

    headers = None
    if token is not None:
        headers = {"Accept": "application/vnd.github+json", "Authorization": f"Bearer {token}"}

    url = f"https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}/jobs?per_page=100"
    result = requests.get(url, headers=headers).json()
    job_time = {}

    try:
        job_time.update({job["name"]: extract_time_from_single_job(job) for job in result["jobs"]})
        pages_to_iterate_over = math.ceil((result["total_count"] - 100) / 100)

        for i in range(pages_to_iterate_over):
            result = requests.get(url + f"&page={i + 2}", headers=headers).json()
            job_time.update({job["name"]: extract_time_from_single_job(job) for job in result["jobs"]})

        return job_time
    except Exception:
        print(f"Unknown error, could not fetch links:\n{traceback.format_exc()}")



SUMMARY:
**Function Summary**

`get_job_time`: Extracts time information for all jobs in a GitHub Actions workflow run.

**Primary Purpose:** Retrieves and extracts time info from multiple GitHub Actions workflow jobs.

**Implementation Details:**

- Uses `requests` to fetch GitHub API data
- Iterates through 2 pages of jobs (due to pagination) to collect time info
- Utilizes `extract_time_from_single_job` function (not shown in this snippet)
- Handles exceptions and prints error messages if an issue occurs

**Design Choices:**

* Uses Bearer token authentication for API requests
* Sets a page size of 100 jobs per request (per GitHub's rate limiting guidelines)

**Notable Dependencies:**

* `requests`
* `math` (for calculating total pages to iterate over)
* `extract_time_from_single_job` (not shown in this snippet, assumed to be implemented elsewhere)


key: chroma:document
string_value:
CODE:

def get_last_stable_minor_release():
    # Get the last stable release of transformers
    url = "https://pypi.org/pypi/transformers/json"
    release_data = requests.get(url).json()

    # Find the last stable release of of transformers (version below current version)
    major_version, minor_version, patch_version, _ = current_version.split(".")
    last_major_minor = f"{major_version}.{int(minor_version) - 1}"
    last_stable_minor_releases = [
        release for release in release_data["releases"] if release.startswith(last_major_minor)
    ]
    last_stable_release = sorted(last_stable_minor_releases, key=version.parse)[-1]



SUMMARY:
**Summary:**

The `get_last_stable_minor_release` function retrieves the last stable minor release of the `transformers` library by querying the PyPI JSON API. It compares the current major version to determine the last stable major and minor version, then filters releases to find those with versions below this threshold.

**Key Implementation Details:**

* Fetches `transformers` Pypi JSON data using `requests`
* Compares current version to find last stable minor release
* Uses `version.parse()` to sort and compare release versions

**Notable Dependencies/Requirements:**

* `requests` library for HTTP client
* `pyperpapi` library (`version.parse()`) for parsing release versions


key: chroma:document
string_value:
CODE:

def build_tip_message(last_stable_release):
    return (
        """
<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
"""
        + f"""If you run into any issues running this model, please reinstall the last version that supported this model: v{last_stable_release}.
You can do so by running the following command: `pip install -U transformers=={last_stable_release}`.

</Tip>"""


SUMMARY:
**Function Summary: build_tip_message**

* **Purpose**: Generates a maintenance mode warning message for a machine learning model, indicating that new PRs changing the code are not accepted.
	+ Primary purpose: Warn users about model maintenance mode and provide instructions to reinstall previous version.

**Key Implementation Details:**
- Uses f-string formatting to insert `last_stable_release` into the message.
- Includes a Git installation command for reusing the last stable version supported by the model (Transformer vX.X).

**Notable Dependencies or Requirements:** None mentioned explicitly.


key: chroma:document
string_value:
CODE:

def insert_tip_to_model_doc(model_doc_path, tip_message):
    tip_message_lines = tip_message.split("\n")

    with open(model_doc_path, "r") as f:
        model_doc = f.read()

    # Add the tip message to the model doc page directly underneath the title
    lines = model_doc.split("\n")

    new_model_lines = []
    for line in lines:
        if line.startswith("# "):
            new_model_lines.append(line)
            new_model_lines.extend(tip_message_lines)
        else:
            new_model_lines.append(line)

    with open(model_doc_path, "w") as f:


SUMMARY:
**Summary:**

`insert_tip_to_model_doc` function appends a tip message below the title section of an existing Markdown document file.

**Key Details:**

* Reads an input Markdown document file (`model_doc_path`) and tip message (`tip_message`)
* Splits the document into lines, separating those with title markers (`# `) from others
* Inserts the tip message lines below the original title lines, preserving blank line separation

**Implementation Notes:**

* Uses `file read/write` operations to modify an external file resource.

**External Dependencies: None specified. No known inherited or implemented interfaces are assumed to be used in this code snippet.**


key: chroma:document
string_value:
CODE:

def get_model_doc_path(model: str) -> Tuple[Optional[str], Optional[str]]:
    # Possible variants of the model name in the model doc path
    model_names = [model, model.replace("_", "-"), model.replace("_", "")]

    model_doc_paths = [REPO_PATH / f"docs/source/en/model_doc/{model_name}.md" for model_name in model_names]

    for model_doc_path, model_name in zip(model_doc_paths, model_names):
        if os.path.exists(model_doc_path):
            return model_doc_path, model_name



SUMMARY:
**Summary:**

This function retrieves the path to an English model documentation file based on the provided `model` name. It checks for three possible variant names and searches for a non-empty documentation directory within the repository.

**Implementation Details:**

* The function returns two optional strings, (`model_doc_path`, `model_name`) if a matching documentation file exists.
* The search considers three variant names:
	1. Original model name
	2. Model name with hyphens instead of underscores
	3. Model name without hyphens
* It utilizes the `os` module for path existence checks.

**Notable Dependencies:**

* `REPO_PATH`: assumed to be a constants or an external variable referencing the repository's root directory.
* `os.path.exists()`
* `zip()` function


key: chroma:document
string_value:
CODE:

def extract_model_info(model):
    model_info = {}
    model_doc_path, model_doc_name = get_model_doc_path(model)
    model_path = REPO_PATH / f"src/transformers/models/{model}"

    if model_doc_path is None:
        print(f"Model doc path does not exist for {model}")
        return None
    model_info["model_doc_path"] = model_doc_path
    model_info["model_doc_name"] = model_doc_name

    if not os.path.exists(model_path):
        print(f"Model path does not exist for {model}")
        return None
    model_info["model_path"] = model_path



SUMMARY:
**Function Summary: extract_model_info**

**Purpose and Functionality:**
This function retrieves information about a given machine learning model, including its documentation path and file location.

**Key Implementation Details:**

* Retrieves `model` using the `get_model_doc_path` function to get its documentation path.
* Returns a dictionary containing the following keys:
	+ `model_doc_path`: Path to the model's documentation.
	+ `model_doc_name`: Name of the model's documentation.

**Notable Dependencies and Requirements:**

* `REPO_PATH`: A constant representing the repository path.
* `get_model_doc_path` function.
* External dependencies: This function relies on the existence of the `get_model_doc_path` function, which itself may depend on other external libraries.


key: chroma:document
string_value:
CODE:

def update_relative_imports(filename, model):
    with open(filename, "r") as f:
        filelines = f.read()

    new_file_lines = []
    for line in filelines.split("\n"):
        if line.startswith("from .."):
            new_file_lines.append(line.replace("from ..", "from ..."))
        else:
            new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Summary**

`update_relative_imports()` function updates relative import paths in a Python file. It replaces `..` with `...` to update absolute imports on all lines. The updated file is written back to the original location.

**Key Implementation Details**

* Replaces `from ..` with `from ...` across all lines to convert relative imports to absolute ones.
* Uses `split("\n")` to split the file into individual lines for processing.

**Notable Dependencies/Requirements**

* None explicit dependencies. However, it's assumed that this function runs in a Python environment with necessary modules and libraries available.


key: chroma:document
string_value:
CODE:

def remove_copied_from_statements(model):
    model_path = REPO_PATH / f"src/transformers/models/{model}"
    for file in os.listdir(model_path):
        if file == "__pycache__":
            continue
        file_path = model_path / file
        with open(file_path, "r") as f:
            file_lines = f.read()

        new_file_lines = []
        for line in file_lines.split("\n"):
            if "# Copied from" in line:
                continue
            new_file_lines.append(line)

        with open(file_path, "w") as f:


SUMMARY:
**Summary**

This function, `remove_copied_from_statements`, removes commented lines starting with "# Copied from" from the specified model files. The main purpose is to clean upTransformer models.

**Key Implementation Details:**

* Iterates over lines in each file in the model directory, skipping lines containing "# Copied from".
* Uses `os` and `re` modules implicitly for listing directory contents and file reading/writing.
* Writes modified line-by-line to new file, omitting lines with the specified comment.

**Notable Dependencies/Requirements:**

* `REPO_PATH`: reference path (relative to current working directory) containing model files.
* `os`: file system operations.
* `re`: regular expression matching used implicitly for pattern searching.


key: chroma:document
string_value:
CODE:

def move_model_files_to_deprecated(model):
    model_path = REPO_PATH / f"src/transformers/models/{model}"
    deprecated_model_path = REPO_PATH / f"src/transformers/models/deprecated/{model}"

    if not os.path.exists(deprecated_model_path):
        os.makedirs(deprecated_model_path)

    for file in os.listdir(model_path):
        if file == "__pycache__":
            continue
        repo.git.mv(f"{model_path}/{file}", f"{deprecated_model_path}/{file}")

        # For deprecated files, we then need to update the relative imports


SUMMARY:
**Summary: Migrate Model Files to Deprecation**

### Purpose and Functionality

Migrates model files from `src/transformers/models/{model}` to `src/transformers/models/deprecated/{model}`.

### Key Implementation Details

* Uses `repo.git.mv` to rename files, updating them at the new location.
* Calls `update_relative_imports` to update relative imports for deprecated files.

### Notable Dependencies or Requirements

* `REPO_PATH`: Base path of the repository
* `os.path.exists`, `os.makedirs`, `repo.git.mv`
* `update_relative_imports`

Note: The exact implementation details may vary based on the `update_relative_imports` function, which is not shown in this snippet.


key: chroma:document
string_value:
CODE:

def delete_model_tests(model):
    tests_path = REPO_PATH / f"tests/models/{model}"

    if os.path.exists(tests_path):


SUMMARY:
**Function Summary: `delete_model_tests`**

 Deletes model test files from the repository, removing related Git tracking.

**Main Purpose and Functionality:**
The primary purpose is to delete model test files at the specified path if they exist. It utilizes the Git command-line interface (`repo.git.rm`) to remove these files and their associated Git tracking.

**Key Implementation Details:**

* Uses `os.path.exists` to check for file existence.
* Utilizes `REPO_PATH / f"tests/models/{model}"` as the test path, referencing the `REPO_PATH` constant.
* Executes `repo.git.rm("-r", tests_path)` to remove the files and their tracking using Git.

**Notable Dependencies or Requirements:**

* References `repo`, an existing Git repository object.
* Utilizes external tool, `os`.
* `REPO_PATH`, a constant referencing the repository path.


key: chroma:document
string_value:
CODE:

def get_line_indent(s):


SUMMARY:
**Summary:**

Calculates the line indentation by subtracting the length of the stripped leading spaces from the total string length.

**Key Implementations:**

* `lstrip()` method is used to remove trailing whitespace characters
* Function takes a single argument `s` (string)

**Notable Dependencies/Requirements:**

* None mentioned

This function is used to calculate and return the indentation line in a given string. The implementation relies on the built-in `lstrip()` method, providing a concise and efficient solution for checking leading whitespace characters.


key: chroma:document
string_value:
CODE:

def update_main_init_file(models):
    """
    Replace all instances of model.model_name with model.deprecated.model_name in the __init__.py file

    Args:
        models (List[str]): The models to mark as deprecated
    """
    filename = REPO_PATH / "src/transformers/__init__.py"
    with open(filename, "r") as f:
        init_file = f.read()

    # 1. For each model, find all the instances of model.model_name and replace with model.deprecated.model_name
    for model in models:
        init_file = init_file.replace(f'models.{model}"', f'models.deprecated.{model}"')
        init_file = init_file.replace(f"models.{model} import", f"models.deprecated.{model} import")

    with open(filename, "w") as f:
        f.write(init_file)

    # 2. Resort the imports


SUMMARY:
**Function Summary: `update_main_init_file`**

**Primary Purpose:** Replaces deprecated model names in `__init__.py` file with new, deprecated versions.

**Key Implementation Details:**

* Replaces instances of `model.model_name` with `model.deprecated.model_name` in the `__init__.py` file.
* Also updates import statements to use the deprecated module name.

**Notable Dependencies and Requirements:**

* `REPO_PATH`: The absolute path to the repository's source directory (`"src/transformers/__init__.py"`).
* Uses an external function `sort_imports_in_all_inits(check_only=False)` (not shown in the provided code snippet).


key: chroma:document
string_value:
CODE:

def remove_model_references_from_file(filename, models, condition):
    """
    Remove all references to the given models from the given file

    Args:
        filename (str): The file to remove the references from
        models (List[str]): The models to remove
        condition (Callable): A function that takes the line and model and returns True if the line should be removed
    """
    filename = REPO_PATH / filename
    with open(filename, "r") as f:
        init_file = f.read()

    new_file_lines = []
    for i, line in enumerate(init_file.split("\n")):
        if any(condition(line, model) for model in models):
            continue
        new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Summary:**

`remove_model_references_from_file` function:

* Purpose: Removes references to specified models from a given file by applying a filter condition.
* Key Implementation Details:
	+ Reads the entire file into memory using `init_file = f.read()`.
	+ Iterates over each line, skipping those that match any model based on the `condition` function.
	+ Writes the new filtered file contents back to disk.
* Notable Dependencies and Requirements:
	+ Uses Python's built-in `open()` and `read/write` functions for file input/output.
	+ Assumes a `REPO_PATH` constant or variable defining the repository root path.
	+ Accepts a `condition` function with signature `Callable[[str, str], bool]`.

**RAG (Reusable Anonymous Code) System:**

```markdown
# remove_model_references_from_file
### Purpose
Removes references to specified models from a given file.

### Key Implementation Details

* Reads entire file into memory.
* Applies filter condition to line-model pairs.
* Writes new filtered file contents back to disk.

### Dependencies and Requirements

* Python built-in `open` and `read/write`.
* `REPO_PATH` constant or variable.
* `condition` function signature: `Callable[[str, str], bool]`.
```


key: chroma:document
string_value:
CODE:

def remove_model_config_classes_from_config_check(model_config_classes):
    """
    Remove the deprecated model config classes from the check_config_attributes.py file

    Args:
        model_config_classes (List[str]): The model config classes to remove e.g. ["BertConfig", "DistilBertConfig"]
    """
    filename = REPO_PATH / "utils/check_config_attributes.py"
    with open(filename, "r") as f:
        check_config_attributes = f.read()

    # Keep track as we have to delete comment above too
    in_special_cases_to_allow = False
    in_indent = False
    new_file_lines = []

    for line in check_config_attributes.split("\n"):
        indent = get_line_indent(line)
        if (line.strip() == "SPECIAL_CASES_TO_ALLOW = {") or (line.strip() == "SPECIAL_CASES_TO_ALLOW.update("):
            in_special_cases_to_allow = True

        elif in_special_cases_to_allow and indent == 0 and line.strip() in ("}", ")"):
            in_special_cases_to_allow = False

        if in_indent:
            if line.strip().endswith(("]", "],")):
                in_indent = False
            continue

        if in_special_cases_to_allow and any(
            model_config_class in line for model_config_class in model_config_classes
        ):
            # Remove comments above the model config class to remove
            while new_file_lines[-1].strip().startswith("#"):
                new_file_lines.pop()

            if line.strip().endswith("["):
                in_indent = True

            continue

        elif any(model_config_class in line for model_config_class in model_config_classes):
            continue

        new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Function Summary:**

`remove_model_config_classes_from_config_check`

Removes deprecated model configuration classes from the `check_config_attributes.py` file.

**Purpose and Functionality:**

* Reads the contents of the specified Python file (`utils/check_config_attributes.py`)
* Iterates through each line, tracking indentation and special cases
* Removes comments above deprecated model config classes and rewrites the content in a new file

**Key Implementation Details:**

* Uses regex-free text processing to identify lines with model config classes
* Preserves indentation and formatting when rewriting the content
* Handles edge cases such as nested brackets and escaped newline characters

**Dependencies and Interfaces:**

* None, but may utilize external `REPO_PATH` constant in a larger context.
* Does not implement or inherit any specific interfaces.

**Method Signatures:**

* `remove_model_config_classes_from_config_check(model_config_classes)`: takes a list of string model config class names as input


key: chroma:document
string_value:
CODE:

def add_models_to_deprecated_models_in_config_auto(models):
    """
    Add the models to the DEPRECATED_MODELS list in configuration_auto.py and sorts the list
    to be in alphabetical order.
    """
    filepath = REPO_PATH / "src/transformers/models/auto/configuration_auto.py"
    with open(filepath, "r") as f:
        config_auto = f.read()

    new_file_lines = []
    deprecated_models_list = []
    in_deprecated_models = False
    for line in config_auto.split("\n"):
        if line.strip() == "DEPRECATED_MODELS = [":
            in_deprecated_models = True
            new_file_lines.append(line)
        elif in_deprecated_models and line.strip() == "]":
            in_deprecated_models = False
            # Add the new models to deprecated models list
            deprecated_models_list.extend([f'    "{model}", ' for model in models])
            # Sort so they're in alphabetical order in the file
            deprecated_models_list = sorted(deprecated_models_list)
            new_file_lines.extend(deprecated_models_list)
            # Make sure we still have the closing bracket
            new_file_lines.append(line)
        elif in_deprecated_models:
            deprecated_models_list.append(line)
        else:
            new_file_lines.append(line)

    with open(filepath, "w") as f:


SUMMARY:
**Function Summary:**

`add_models_to_deprecated_models_in_config_auto(models)`

Adds the input `models` to the list of deprecated models in `configuration_auto.py`, updating its contents and sorting the list alphabetically.

**Key Implementation Details:**

- Reads `DEPRECATED_MODELS` from `configuration_auto.py`.
- Appends new models to the list in alphabetical order.
- Updates the file with the modified content.

**Notable Dependencies/Requirements:**

- `REPO_PATH`: reference to a repository-specific path ( exact class/function name required)
- External dependency on:
  - `os` module for file system operations
  - `pathlib` module for handling file paths
  - `shlex` module (not explicitly used but possibly implied by the `split()` and `strip()` functions)

**Interface/Import Notes:**

- No explicit import statements are provided, suggesting that necessary modules and classes will be imported within this code.


key: chroma:document
string_value:
CODE:

def deprecate_models(models):
    # Get model info
    skipped_models = []
    models_info = defaultdict(dict)
    for model in models:
        single_model_info = extract_model_info(model)
        if single_model_info is None:
            skipped_models.append(model)
        else:
            models_info[model] = single_model_info

    model_config_classes = []
    for model, model_info in models_info.items():
        if model in CONFIG_MAPPING:
            model_config_classes.append(CONFIG_MAPPING[model].__name__)
        elif model_info["model_doc_name"] in CONFIG_MAPPING:
            model_config_classes.append(CONFIG_MAPPING[model_info["model_doc_name"]].__name__)
        else:
            skipped_models.append(model)
            print(f"Model config class not found for model: {model}")

    # Filter out skipped models
    models = [model for model in models if model not in skipped_models]

    if skipped_models:
        print(f"Skipped models: {skipped_models} as the model doc or model path could not be found.")
    print(f"Models to deprecate: {models}")

    # Remove model config classes from config check
    print("Removing model config classes from config checks")
    remove_model_config_classes_from_config_check(model_config_classes)

    tip_message = build_tip_message(get_last_stable_minor_release())

    for model, model_info in models_info.items():
        print(f"Processing model: {model}")
        # Add the tip message to the model doc page directly underneath the title
        print("Adding tip message to model doc page")
        insert_tip_to_model_doc(model_info["model_doc_path"], tip_message)

        # Remove #Copied from statements from model's files
        print("Removing #Copied from statements from model's files")
        remove_copied_from_statements(model)

        # Move the model file to deprecated: src/transfomers/models/model -> src/transformers/models/deprecated/model
        print("Moving model files to deprecated for model")
        move_model_files_to_deprecated(model)

        # Delete the model tests: tests/models/model
        print("Deleting model tests")
        delete_model_tests(model)

    # # We do the following with all models passed at once to avoid having to re-write the file multiple times
    print("Updating __init__.py file to point to the deprecated models")
    update_main_init_file(models)

    # Remove model references from other files
    print("Removing model references from other files")
    remove_model_references_from_file(
        "src/transformers/models/__init__.py", models, lambda line, model: model == line.strip().strip(",")
    )
    remove_model_references_from_file(
        "utils/slow_documentation_tests.txt", models, lambda line, model: "/" + model + "/" in line
    )
    remove_model_references_from_file("utils/not_doctested.txt", models, lambda line, model: "/" + model + "/" in line)

    # Add models to DEPRECATED_MODELS in the configuration_auto.py
    print("Adding models to DEPRECATED_MODELS in configuration_auto.py")


SUMMARY:
**Function Summary: deprecate_models**

### Purpose and Functionality:

The `deprecate_models` function is designed to deprecate specific models by updating their documentation, removing references, and refactoring their configurations. It iterates through a list of models, extracting information about each model from an `extract_model_info` function call. The function checks if the model has a configured class in the `CONFIG_MAPPING`, adds a tip message to the model's documentation page, removes redundant statements, moves the model files to the deprecated directory, deletes test files, and updates other files that reference the models.

### Key Implementation Details:

*   Iterates through a list of models using `models` as input.
*   Calls `extract_model_info(model)` for each model to gather information.
*   Checks if a configured class exists in `CONFIG_MAPPING` or by name within `CONFIG_MAPPING`.
*   Updates the last stable minor release version using `get_last_stable_minor_release()`.
*   Uses `build_tip_message(tip_message)`, `remove_model_config_classes_from_config_check(model_config_classes)`, and other functions to finalize model depreciation.
*   Iterates over models, processing each one individually using `insert_tip_to_model_doc`, `remove_copied_from_statements`, `move_model_files_to_deprecated`, `delete_model_tests`, and updates files like `update_main_init_file` to point to the deprecated models.

### Dependencies and Requirements:

*   Uses `defaultdict` from Python's standard library for creating default dictionaries.
*   References `CONFIG_MAPPING` (a dictionary of model configurations) at least once during execution, which must be defined elsewhere in the codebase.
*   Calls functions with external dependencies like `remove_model_references_from_file`, `update_main_init_file`, and others that reference files or classes not specified within this function.

### Notable Interfaces:

The function provides a few notable interfaces to other components of the system:
    
*   `CONFIG_MAPPING`: A dictionary mapping model names or documentation lines to their respective configuration classes.
*   External dependencies such as `update_main_init_file`, `remove_model_references_from_file` (referencing specific files or directories), and `build_tip_message`.


key: chroma:document
string_value:
CODE:

def sort_auto_mapping(fname: str, overwrite: bool = False) -> Optional[bool]:
    """
    Sort all auto mappings in a file.

    Args:
        fname (`str`): The name of the file where we want to sort auto-mappings.
        overwrite (`bool`, *optional*, defaults to `False`): Whether or not to fix and overwrite the file.

    Returns:
        `Optional[bool]`: Returns `None` if `overwrite=True`. Otherwise returns `True` if the file has an auto-mapping
        improperly sorted, `False` if the file is okay.
    """
    with open(fname, "r", encoding="utf-8") as f:
        content = f.read()

    lines = content.split("\n")
    new_lines = []
    line_idx = 0
    while line_idx < len(lines):
        if _re_intro_mapping.search(lines[line_idx]) is not None:
            # Start of a new mapping!
            indent = len(re.search(r"^(\s*)\S", lines[line_idx]).groups()[0]) + 8
            while not lines[line_idx].startswith(" " * indent + "("):
                new_lines.append(lines[line_idx])
                line_idx += 1

            blocks = []
            while lines[line_idx].strip() != "]":
                # Blocks either fit in one line or not
                if lines[line_idx].strip() == "(":
                    start_idx = line_idx
                    while not lines[line_idx].startswith(" " * indent + ")"):
                        line_idx += 1
                    blocks.append("\n".join(lines[start_idx : line_idx + 1]))
                else:
                    blocks.append(lines[line_idx])
                line_idx += 1

            # Sort blocks by their identifiers
            blocks = sorted(blocks, key=lambda x: _re_identifier.search(x).groups()[0])
            new_lines += blocks
        else:
            new_lines.append(lines[line_idx])
            line_idx += 1

    if overwrite:
        with open(fname, "w", encoding="utf-8") as f:
            f.write("\n".join(new_lines))
    else:


SUMMARY:
**Function Summary: `sort_auto_mapping`**

The `sort_auto_mapping` function sorts the auto-mappings in a given file.

### Purpose and Functionality:

Automatically rearranges indented lines containing mappings in the specified file to adhere to a specific format. Compares and returns whether the original formatting is preserved upon sorting (when `overwrite=False`) or if changes were made for overwriting (`overwrite=True`).

### Key Implementation Details:

1.  **Regex-based Pattern Matching**:
    *   `_re_intro_mapping.search()`: Identifies the start of a new mapping in lines.
2.  **Blocks and Lines Processing**:
    *   `indentation calculation`: Calculates indentation length before processing each mapping block.
    *   `block decomposition` (`if lines[line_idx].strip() == "(":`): Tackles blocks divided across multiple lines with indentation matching the first line's formatting.
3.  **Sorting Algorithm**:
    *   `blocks sorted by ids` (`sorted(blocks, key=lambda x: _re_identifier.search(x).groups()[0])`): Orders and groups mappings according to their unique identifiers.

### Notable Dependencies or Requirements:

1.  `re` module (regular expressions) for pattern matching.
2.  External file operations via `open` function with UTF-8 encoding.

**API Reference:**

*   `_re_intro_mapping`, `_re_identifier`: Externally defined regular expression patterns used within the sorting logic, not part of this summary but crucial for implementation context.

### Example Usage:
```python
result = sort_auto_mapping('path/to/file.txt')
print(result)  # Returns 'None' if overwrite is False and file has been properly sorted, False otherwise.
```


