# embedding_metadata

key: chroma:document
string_value:
CODE:

def pytest_configure(config):
    config.addinivalue_line(
        "markers", "is_pt_tf_cross_test: mark test to run only when PT and TF interactions are tested"
    )
    config.addinivalue_line(
        "markers", "is_pt_flax_cross_test: mark test to run only when PT and FLAX interactions are tested"
    )
    config.addinivalue_line("markers", "is_pipeline_test: mark test to run only when pipelines are tested")
    config.addinivalue_line("markers", "is_staging_test: mark test to run only in the staging environment")
    config.addinivalue_line("markers", "accelerate_tests: mark test that require accelerate")
    config.addinivalue_line("markers", "agent_tests: mark the agent tests that are run on their specific schedule")


SUMMARY:
**RAG Code Summary**

Function Name: `pytest_configure`

Purpose:
Enable and define custom markers for pytest tests in the configuration.

Key Implementation Details:

* Add custom marker lines with descriptive names, e.g. `is_pt_tf_cross_test`, `accelerate_tests`
* Define a total of 7 unique marker settings

Notable Dependencies or Requirements:
- Pytest library for testing framework
- N/A (only internal pytest configuration)

Methods and Interfaces Used:

None

External Dependencies:
- None

Design Choices:

- Utilizing `config.addinivalue_line` to extend pytest configuration
- Employing custom markers for specific testing scenarios

Primary Purpose: Enhance pytest test execution by providing customizable testing flag configurations.


key: chroma:document
string_value:
CODE:

def pytest_collection_modifyitems(items):
    for item in items:
        if any(test_name in item.nodeid for test_name in NOT_DEVICE_TESTS):


SUMMARY:
**Summary:**

`pytest_collection_modifyitems` is a function that modifies a list of test items. It filters out items whose names contain specific device-related tests (`NOT_DEVICE Tests`) and adds the `not_device_test` marker to remaining items.

**Key Details:**

* Filters test items by name using `any(check in item.nodeid for check in NOT_DEVICE_TESTS)`
* Adds `pytest.mark.not_device_test` marker to filtered items
* No specific algorithm or pattern used, but relies on Python's built-in string matching logic

**Implementation Dependencies:**

* `pytest`

**Interfaces/Inheritance:**

* Implements the ` pytest.Collection` interface


key: chroma:document
string_value:
CODE:

def pytest_addoption(parser):
    from transformers.testing_utils import pytest_addoption_shared



SUMMARY:
**Function Summary**

`pytest_addoption`: A custom pytest hook to add options for testing Hugging Face model transformers.

**Key Implementation Details:**

* Calls `pytest_addoption_shared` from `transformers.testing_utils.pytest` with the same parser instance.

**Notable Dependencies/Requirements:**

* `pytest.testing_utils`: Provides a shared implementation for adding options in pytest.
* No external dependencies specified.


key: chroma:document
string_value:
CODE:

def pytest_terminal_summary(terminalreporter):
    from transformers.testing_utils import pytest_terminal_summary_main

    make_reports = terminalreporter.config.getoption("--make-reports")
    if make_reports:


SUMMARY:
**Summary: Pytest Terminal Summary Function**

The `pytest_terminal_summary` function is a custom pytest fixture that determines whether to generate terminal reports. When the "--make-reports" option is specified, it calls the `pytest_terminal_summary_main` function with the provided reporter.

**Key Details:**

* Primary purpose: Determine whether to generate terminal reports based on the "--make-reports" option.
* Implementation: Checks for the existence of the "--make-reports" option and uses it as input for the `pytest_terminal_summary_main` function.

**Dependencies/Notable Requirements:**

* `terminalreporter`: The input reporter used by pytest-terminal-summary.
* `pytransformers`: A library containing the `pytest_terminal_summary_main` function.


key: chroma:document
string_value:
CODE:

def pytest_sessionfinish(session, exitstatus):
    # If no tests are collected, pytest exists with code 5, which makes the CI fail.
    if exitstatus == 5:


SUMMARY:
**Function Summary:**

`pytest_sessionfinish`: A fixture function executed when all tests in a Pytest session have finished execution.

**Primary Purpose:** Prevents Pytest from reporting an error code of 5, indicating no tests were collected, and instead sets the exit status to successful (0).

**Key Implementation Details:**

* Sets `session.exitstatus` to 0 if the current test session has an unknown or undefined existence code of 5 (`exitstatus == 5`)

**Notable Dependencies/Requirements:**

* Pytest framework
* `pytest ExitStatusError` exception


key: chroma:document
string_value:
CODE:

class CustomOutputChecker(OutputChecker):
    def check_output(self, want, got, optionflags):
        if IGNORE_RESULT & optionflags:
            return True


SUMMARY:
**Summary**

### Class Overview

`CustomOutputChecker` extends the `OutputChecker` class and overrides its `check_output` method.

### Primary Purpose

Customizes the behavior of output checking based on specific option flags (`optionflags`).

### Key Implementation Details

* Uses bitwise operator to check for presence of `IGNORE_RESULT` flag in `optionflags`, returning True if it's present.
* Calls the inherited `OutputChecker.check_output` method with the same arguments.

### Notable Dependencies/Requirements

* Extends `OutputChecker` class
* Imports `OutputChecker` and uses its `check_output` method directly.


key: docstring
string_value:
# since we save this data in src/transformers/dependency_versions_table.py it can be easily accessed from
# anywhere. If you need to quickly access the data from this table in a shell, you can do so easily with:
#
# python -c 'import sys; from transformers.dependency_versions_table import deps; \
# print(" ".join([ deps[x] for x in sys.argv[1:]]))' tokenizers datasets
#
# Just pass the desired package names to that script as it's shown with 2 packages above.
#
# If transformers is not yet installed and the work is done from the cloned repo remember to add `PYTHONPATH=src` to the script above
#
# You can then feed this for example to `pip`:
#
# pip install -U $(python -c 'import sys; from transformers.dependency_versions_table import deps; \
# print(" ".join([deps[x] for x in sys.argv[1:]]))' tokenizers datasets)
#


key: chroma:document
string_value:
CODE:

def deps_list(*pkgs):


SUMMARY:
**Function Summary**

`deps_list` function generates a list of dependencies for a given set of packages (`*pkgs`) by retrieving the dependency information from `dependency_versions_table.py`.

**Key Implementation Details**

* List comprehension to generate dependency dependencies in `[deps[pkg] for pkg in pkgs]`.
* External dependency or interface: none.
* Notable design choices:

  * Stores data in a file (`dependency_versions_table.py`) for easy access.
  * Allows external execution using Python shell script.

**Primary Purpose**

Generates list of dependencies for specified packages.


key: chroma:document
string_value:
CODE:

class DepsTableUpdateCommand(Command):
    """
    A custom distutils command that updates the dependency table.
    usage: python setup.py deps_table_update
    """

    description = "build runtime dependency table"
    user_options = [
        # format: (long option, short option, description).
        ("dep-table-update", None, "updates src/transformers/dependency_versions_table.py"),
    ]

    def initialize_options(self):
        pass

    def finalize_options(self):
        pass

    def run(self):
        entries = "\n".join([f'    "{k}": "{v}",' for k, v in deps.items()])
        content = [
            "# THIS FILE HAS BEEN AUTOGENERATED. To update:",
            "# 1. modify the `_deps` dict in setup.py",
            "# 2. run `make deps_table_update``",
            "deps = {",
            entries,
            "}",
            "",
        ]
        target = "src/transformers/dependency_versions_table.py"
        print(f"updating {target}")
        with open(target, "w", encoding="utf-8", newline="\n") as f:


SUMMARY:
**Summary:**

The `DepsTableUpdateCommand` class is a custom distutils command that updates the dependency table. It parses the `_deps` dictionary and generates a runtime dependency table by creating a new file `src/transformers/dependency_versions_table.py`.

**Key Implementation Details:**

* Generates content for a Python file at `src/transformers/dependency_versions_table.py`
* Updates the file with pre-defined header comment
* Writes the updated `_deps` dictionary to the file

**Notable Dependencies and Requirements:**

* None explicit dependencies listed
* Implicit dependency on `setup.py` and `make` utility

**Method Signatures and Interfaces:**

* `initialize_options()`: an empty method used for initialization purposes
* `finalize_options()`: an empty method used for finalization purposes
* `run()`: the main entry point that generates the content for the file


key: chroma:document
string_value:
CODE:

def parse_pytest_output(file_path):
    skipped_tests = {}
    skipped_count = 0
    with open(file_path, 'r') as file:
        for line in file:
            match = re.match(r'^SKIPPED \[(\d+)\] (tests/.*): (.*)$', line)
            if match:
                skipped_count += 1
                test_file, test_line, reason = match.groups()
                skipped_tests[reason] = skipped_tests.get(reason, []) + [(test_file, test_line)]
    for k,v in sorted(skipped_tests.items(), key=lambda x:len(x[1])):
        print(f"{len(v):4} skipped because: {k}")


SUMMARY:
**Summary**

The `parse_pytest_output` function parses the output of a Pytest test run from a files specified by `file_path`. It extracts skipped tests, counting and printing their reasons, along with the number of skipped tests.

**Main Purpose and Functionality**

*   Extracts skipped tests from a Pytest output file
*   Counts and prints each reason for skipping along with its corresponding test files and lines

**Key Implementation Details**

*   Uses regular expressions to match `SKIPPED` test markers (`'^SKIPPED \[(\d+)\] (tests/.*): (.*)$'`)
*   Iterates through the Pytest output file, processing each line for skipped tests
*   Uses a dictionary (`skipped_tests`) to store skipping reasons as keys and lists of corresponding tests as values

**Notable Dependencies or Requirements**

*   Utilizes `re` module for regular expression matching
*   Implies usage of the `sorted()` function for grouping sorted skips by reason length


key: chroma:document
string_value:
CODE:

def parse_pytest_failure_output(file_path):
    failed_tests = {}
    failed_count = 0
    with open(file_path, 'r') as file:
        for line in file:
            match = re.match(r'^FAILED (tests/.*) - (.*): (.*)$', line)
            if match:
                failed_count += 1
                _, error, reason = match.groups()
                failed_tests[reason] = failed_tests.get(reason, []) + [error]
    for k,v in sorted(failed_tests.items(), key=lambda x:len(x[1])):
        print(f"{len(v):4} failed because `{v[0]}` -> {k}")
    print("Number of failed tests:", failed_count)
    if failed_count>0:


SUMMARY:
**Function Summary: `parse_pytest_failure_output`**

**Purpose and Functionality:**
This function parses the output of pytest failure reports from a text file, extracting test cases and their corresponding error messages.

**Key Implementation Details:**

* Uses regular expression to match failure report lines, which follow the format: 'FAILED (tests/<test_name>): (error message); (reason for failure)'
* Iterates through the file, incrementing a count of failed tests and storing error messages in a dictionary (`failed_tests`) keyed by reason.
* Prints the top reasons with the most errors, alongside their frequencies and corresponding error messages.

**Notable Dependencies or Requirements:**

* `re` (regular expression) module for pattern matching
* No other external dependencies explicitly mentioned

Note: The function exits if any tests failed (return code 1).


key: chroma:document
string_value:
CODE:

def parse_pytest_errors_output(file_path):
    print(file_path)
    error_tests = {}
    error_count = 0
    with open(file_path, 'r') as file:
        for line in file:
            match = re.match(r'^ERROR (tests/.*) - (.*): (.*)$', line)
            if match:
                error_count += 1
                _, test_error, reason = match.groups()
                error_tests[reason] = error_tests.get(reason, []) + [test_error]
    for k,v in sorted(error_tests.items(), key=lambda x:len(x[1])):
        print(f"{len(v):4} errored out because of `{v[0]}` -> {k}")
    print("Number of errors:", error_count)
    if error_count>0:


SUMMARY:
**Function Summary: `parse_pytest_errors_output`**

### Main Purpose and Functionality

This function parses the output of pytest to collect error test cases. It reads a file at `file_path`, extracts error test cases from lines matching the `ERROR` pattern, and prints the error frequency along with reasons and corresponding test errors.

**Key Implementation Details**

* Uses regular expression (`re.match`) to extract error message components.
* Groups extracted fields into test errors for analysis (e.g., `error_tests[reason] = ...`).
* Sorts test errors alphabetically by reason before printing results.

**Notable Dependencies or Requirements**

* `re` module for regular expressions
* External file at `file_path` containing pytest output


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--file", help="file to parse")
    parser.add_argument("--skip", action="store_true", help="show skipped reasons")
    parser.add_argument("--fail", action="store_true", help="show failed tests")
    parser.add_argument("--errors", action="store_true", help="show failed tests")
    args = parser.parse_args()

    if args.skip:
        parse_pytest_output(args.file)

    if args.fail:
        parse_pytest_failure_output(args.file)

    if args.errors:


SUMMARY:
**Summary**

This `main` function defines an argument parser to handle parsing file options. It takes the following arguments:

* `--file`: input file to be parsed
* `--skip`: show skipped reasons
* `--fail`: show failed tests
* `--errors`: show parsing errors (not `--fail`, as in different terminology)

The function parses the command-line arguments and then calls separate functions `parse_pytest_output`, `parse_pytest_failure_output`, and `parse_pytest_errors_output` based on the input parameters. These functions are not shown in this code snippet.

Key implementation details:

* Uses Argparse to define a parser with file-specific arguments
* Parses command-line arguments using `parser.parse_args()`
* Calls separate parsing functions based on input parameters

Notable dependencies or requirements:

* Argparse library for parsing command-line arguments (`argparse.ArgumentParser`)
* Custom functions: `parse_pytest_output`, `parse_pytest_failure_output`, and `parse_pytest_errors_output` (not shown)


key: chroma:document
string_value:
CODE:

class EmptyJob:
    job_name = "empty"

    def to_dict(self):
        steps = [{"run": 'ls -la'}]
        if self.job_name == "collection_job":
            steps.extend(
                [
                    "checkout",
                    {"run": "pip install requests || true"},
                    {"run": """while [[ $(curl --location --request GET "https://circleci.com/api/v2/workflow/$CIRCLE_WORKFLOW_ID/job" --header "Circle-Token: $CCI_TOKEN"| jq -r '.items[]|select(.name != "collection_job")|.status' | grep -c "running") -gt 0 ]]; do sleep 5; done || true"""},
                    {"run": 'python utils/process_circleci_workflow_test_reports.py --workflow_id $CIRCLE_WORKFLOW_ID || true'},
                    {"store_artifacts": {"path": "outputs"}},
                    {"run": 'echo "All required jobs have now completed"'},
                ]
            )

        return {
            "docker": copy.deepcopy(DEFAULT_DOCKER_IMAGE),
            "resource_class": "small",
            "steps": steps,


SUMMARY:
**Summary**

The `EmptyJob` class creates a CI/CD workflow job. Its primary purpose is to simulate a collection job with multiple parallelized steps.

**Key Implementation Details:**

*   **Steps and Script:** The job has two main execution paths:
    *   A simple "ls -la" command.
    *   A more complex script that includes:
        *   Checking if the specified CircleCI job is running.
        *   Installing required packages using pip (requests).
        *   Running a Python script to process test reports.
    This script requires external dependencies, including `circleci.com` and `requests`.

**Dependencies:**

*   External dependency on `circleci.com` for API access:
    +   `$CIRCLE_WORKFLOW_ID`
    +   `$CCI_TOKEN`
*   External dependency on pip for package installation (`requests`)
*   Custom class `utils/process_circleci_workflow_test_reports.py` for processing test reports

**Notable Features:**

*   Modular execution using parallelized steps with conditional logic
*   Involves external interactions with CircleCI API and pip package manager


key: chroma:document
string_value:
CODE:

def create_circleci_config(folder=None):
    if folder is None:
        folder = os.getcwd()
    os.environ["test_preparation_dir"] = folder
    jobs = [k for k in ALL_TESTS if os.path.isfile(os.path.join("test_preparation" , f"{k.job_name}_test_list.txt") )]
    print("The following jobs will be run ", jobs)

    if len(jobs) == 0:
        jobs = [EmptyJob()]
    else:
        print("Full list of job name inputs", {j.job_name + "_test_list":{"type":"string", "default":''} for j in jobs})
        # Add a job waiting all the test jobs and aggregate their test summary files at the end
        collection_job = EmptyJob()
        collection_job.job_name = "collection_job"
        jobs = [collection_job] + jobs

    config = {
        "version": "2.1",
        "parameters": {
            # Only used to accept the parameters from the trigger
            "nightly": {"type": "boolean", "default": False},
            "tests_to_run": {"type": "string", "default": ''},
            **{j.job_name + "_test_list":{"type":"string", "default":''} for j in jobs},
            **{j.job_name + "_parallelism":{"type":"integer", "default":1} for j in jobs},
        },
        "jobs": {j.job_name: j.to_dict() for j in jobs}
    }
    if "CIRCLE_TOKEN" in os.environ:
        # For private forked repo. (e.g. new model addition)
        config["workflows"] = {"version": 2, "run_tests": {"jobs": [{j.job_name: {"context": ["TRANSFORMERS_CONTEXT"]}} for j in jobs]}}
    else:
        # For public repo. (e.g. `transformers`)
        config["workflows"] = {"version": 2, "run_tests": {"jobs": [j.job_name for j in jobs]}}
    with open(os.path.join(folder, "generated_config.yml"), "w") as f:


SUMMARY:
**Function Summary**

The `create_circleci_config` function generates a CI configuration file for CircleCI. It takes an optional `folder` parameter, defaulting to the current working directory. The function constructs a configuration object with parameters and jobs, which is then written to a YAML file.

**Key Implementation Details**

* Iterates over `ALL_TESTS` job names and selects only those with corresponding test files.
* Adds a "collection_job" that waits for all other jobs to finish.
* Constructs the `config` dictionary with:
	+ Parameters for nightly builds and test summaries.
	+ Jobs as dictionaries, populated from their respective `to_dict()` methods.

**Notable Dependencies**

* `os`, `ALL_TESTS` (custom data structure containing job names).
* `yaml`.
* `CircleCI` configuration file formatting conventions (~ pipeline transformations).

**Interfaces and Inherited Dependencies**

* None explicit dependencies.
* Assumes access to `ALL_TESTS` and CircleCI-specific formatting conventions.


key: chroma:document
string_value:
CODE:

class ModelCard:
    r"""
    Structured Model Card class. Store model card as well as methods for loading/downloading/saving model cards.

    Please read the following paper for details and explanation on the sections: "Model Cards for Model Reporting" by
    Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer,
    Inioluwa Deborah Raji and Timnit Gebru for the proposal behind model cards. Link: https://arxiv.org/abs/1810.03993

    Note: A model card can be loaded and saved to disk.
    """

    def __init__(self, **kwargs):
        warnings.warn(
            "The class `ModelCard` is deprecated and will be removed in version 5 of Transformers", FutureWarning
        )
        # Recommended attributes from https://arxiv.org/abs/1810.03993 (see papers)
        self.model_details = kwargs.pop("model_details", {})
        self.intended_use = kwargs.pop("intended_use", {})
        self.factors = kwargs.pop("factors", {})
        self.metrics = kwargs.pop("metrics", {})
        self.evaluation_data = kwargs.pop("evaluation_data", {})
        self.training_data = kwargs.pop("training_data", {})
        self.quantitative_analyses = kwargs.pop("quantitative_analyses", {})
        self.ethical_considerations = kwargs.pop("ethical_considerations", {})
        self.caveats_and_recommendations = kwargs.pop("caveats_and_recommendations", {})

        # Open additional attributes
        for key, value in kwargs.items():
            try:
                setattr(self, key, value)
            except AttributeError as err:
                logger.error(f"Can't set {key} with value {value} for {self}")
                raise err

    def save_pretrained(self, save_directory_or_file):
        """Save a model card object to the directory or file `save_directory_or_file`."""
        if os.path.isdir(save_directory_or_file):
            # If we save using the predefined names, we can load using `from_pretrained`
            output_model_card_file = os.path.join(save_directory_or_file, MODEL_CARD_NAME)
        else:
            output_model_card_file = save_directory_or_file

        self.to_json_file(output_model_card_file)
        logger.info(f"Model card saved in {output_model_card_file}")

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
        r"""
        Instantiate a [`ModelCard`] from a pre-trained model model card.

        Parameters:
            pretrained_model_name_or_path: either:

                - a string, the *model id* of a pretrained model card hosted inside a model repo on huggingface.co.
                - a path to a *directory* containing a model card file saved using the [`~ModelCard.save_pretrained`]
                  method, e.g.: `./my_model_directory/`.
                - a path or url to a saved model card JSON *file*, e.g.: `./my_model_directory/modelcard.json`.

            cache_dir: (*optional*) string:
                Path to a directory in which a downloaded pre-trained model card should be cached if the standard cache
                should not be used.

            kwargs: (*optional*) dict: key/value pairs with which to update the ModelCard object after loading.

                - The values in kwargs of any keys which are model card attributes will be used to override the loaded
                  values.
                - Behavior concerning key/value pairs whose keys are *not* model card attributes is controlled by the
                  *return_unused_kwargs* keyword parameter.

            proxies: (*optional*) dict, default None:
                A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128',
                'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.

            return_unused_kwargs: (*optional*) bool:

                - If False, then this function returns just the final model card object.
                - If True, then this functions returns a tuple *(model card, unused_kwargs)* where *unused_kwargs* is a
                  dictionary consisting of the key/value pairs whose keys are not model card attributes: ie the part of
                  kwargs which has not been used to update *ModelCard* and is otherwise ignored.

        Examples:

        ```python
        # Download model card from huggingface.co and cache.
        modelcard = ModelCard.from_pretrained("google-bert/bert-base-uncased")
        # Model card was saved using *save_pretrained('./test/saved_model/')*
        modelcard = ModelCard.from_pretrained("./test/saved_model/")
        modelcard = ModelCard.from_pretrained("./test/saved_model/modelcard.json")
        modelcard = ModelCard.from_pretrained("google-bert/bert-base-uncased", output_attentions=True, foo=False)
        ```"""
        cache_dir = kwargs.pop("cache_dir", None)
        proxies = kwargs.pop("proxies", None)
        return_unused_kwargs = kwargs.pop("return_unused_kwargs", False)
        from_pipeline = kwargs.pop("_from_pipeline", None)

        user_agent = {"file_type": "model_card"}
        if from_pipeline is not None:
            user_agent["using_pipeline"] = from_pipeline

        is_local = os.path.isdir(pretrained_model_name_or_path)
        if os.path.isfile(pretrained_model_name_or_path):
            resolved_model_card_file = pretrained_model_name_or_path
            is_local = True
        else:
            try:
                # Load from URL or cache if already cached
                resolved_model_card_file = cached_file(
                    pretrained_model_name_or_path,
                    filename=MODEL_CARD_NAME,
                    cache_dir=cache_dir,
                    proxies=proxies,
                    user_agent=user_agent,
                )
                if is_local:
                    logger.info(f"loading model card file {resolved_model_card_file}")
                else:
                    logger.info(f"loading model card file {MODEL_CARD_NAME} from cache at {resolved_model_card_file}")
                # Load model card
                modelcard = cls.from_json_file(resolved_model_card_file)

            except (EnvironmentError, json.JSONDecodeError):
                # We fall back on creating an empty model card
                modelcard = cls()

        # Update model card with kwargs if needed
        to_remove = []
        for key, value in kwargs.items():
            if hasattr(modelcard, key):
                setattr(modelcard, key, value)
                to_remove.append(key)
        for key in to_remove:
            kwargs.pop(key, None)

        logger.info(f"Model card: {modelcard}")
        if return_unused_kwargs:
            return modelcard, kwargs
        else:
            return modelcard

    @classmethod
    def from_dict(cls, json_object):
        """Constructs a `ModelCard` from a Python dictionary of parameters."""
        return cls(**json_object)

    @classmethod
    def from_json_file(cls, json_file):
        """Constructs a `ModelCard` from a json file of parameters."""
        with open(json_file, "r", encoding="utf-8") as reader:
            text = reader.read()
        dict_obj = json.loads(text)
        return cls(**dict_obj)

    def __eq__(self, other):
        return self.__dict__ == other.__dict__

    def __repr__(self):
        return str(self.to_json_string())

    def to_dict(self):
        """Serializes this instance to a Python dictionary."""
        output = copy.deepcopy(self.__dict__)
        return output

    def to_json_string(self):
        """Serializes this instance to a JSON string."""
        return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"

    def to_json_file(self, json_file_path):
        """Save this instance to a json file."""
        with open(json_file_path, "w", encoding="utf-8") as writer:


SUMMARY:
**ModelCard Class Summary**
==========================

**Class Purpose and Functionality**

The `ModelCard` class stores model card data and provides methods for loading, saving, and updating the model card. It is structured around the concept of model cards as described in the paper "Model Cards for Model Reporting" by Margaret Mitchell et al.

**Key Implementation Details**

* The class initializes with attributes such as `model_details`, `intended_use`, and `metrics`.
* The `save_pretrained` method saves the model card object to a directory or file.
* The `from_pretrained` class method instantiates a model card from a pre-trained model name or path.

**Class Method Signatures and Interfaces**

* `__init__`: Initializes the model card object with attributes
* `to_json_file`: Serializes the instance to a JSON string
* `from_dict`, `from_json_file`, and `from_pretrained`: Construct model cards from dictionaries, JSON files, or pre-trained models

**Notable Dependencies and Requirements**

* Python 3.x for compatibility
* `json` library for serializing data
* `cached_file` function from the `cached.io` library (not shown in the code snippet)

**Class Representation**

The class provides an instance of a model card as a dictionary of attributes, which can be serialized to a JSON string and saved to a file. It also offers methods for loading and updating model cards from pre-trained models.


key: chroma:document
string_value:
CODE:

def _listify(obj):
    if obj is None:
        return []
    elif isinstance(obj, str):
        return [obj]
    else:


SUMMARY:
**Summary**

The `_listify` function converts an input object into a list. It handles the following cases:

*   `None`: returns an empty list `[]`
*   `str`: wraps the string in a list `[obj]`
*   Other types: returns the object as is (e.g., integers, lists)

**Key Implementation Details**

*   Returns a specific type of data structure (`list`) based on input type
*   Uses `isinstance` to perform type checks

**Notable Dependencies or Requirements**

*   None


key: chroma:document
string_value:
CODE:

def _insert_values_as_list(metadata, name, values):
    if values is None:
        return metadata
    if isinstance(values, str):
        values = [values]
    values = [v for v in values if v is not None]
    if len(values) == 0:
        return metadata
    metadata[name] = values


SUMMARY:
**Function Summary: `_insert_values_as_list`**

Inserts values into the provided metadata and returns the updated metadata.

**Main Purpose**: Stores non-null values in the `metadata` dictionary under the specified `name`.

**Key Implementation Details**:

- Verifies input data integrity by filtering out `None` values.
- Enforces list-like structure for `values` input, converting it to a list if necessary.
- Adds values to the metadata using dictionary assignment.

**Notable Dependencies/Interfaces**:

* Uses `metadata` dictionary with exact key access (e.g., `metadata[name]`).
* Relying on conditional checks and iterated operations.


key: chroma:document
string_value:
CODE:

def infer_metric_tags_from_eval_results(eval_results):
    if eval_results is None:
        return {}
    result = {}
    for key in eval_results.keys():
        if key.lower().replace(" ", "_") in METRIC_TAGS:
            result[key.lower().replace(" ", "_")] = key
        elif key.lower() == "rouge1":
            result["rouge"] = key


SUMMARY:
**Infer Metric Tags from Evaluation Results**

*Purpose: Reconstructs metric tags from evaluation results for inference purposes.*

**Primary Purpose:**
Infer metric tags by matching keys in `eval_results` with known metric tags in `METRIC_TAGS`.

**Key Implementation Details:**

* Iterates through keys in `eval_results` and checks if any are matched against `METRIC_TAGS`.
* If a match is found, assigns the original key (before whitespace removal) to the result dictionary with modified key (after whitespace removal).
* Special case for "rouge1" key, which maps to "rouge".

**Notable Dependencies/Requirements:**

* `METRIC_TAGS`: a list of known metric tags (not shown in this code snippet).
* No external dependencies listed.

**Code Structure and Design Choices:**

* Simple iterative approach using Python dictionary iteration.
* Efficient use of modified keys for mapping results, reducing potential issues with key conflicts.


key: chroma:document
string_value:
CODE:

def _insert_value(metadata, name, value):
    if value is None:
        return metadata
    metadata[name] = value


SUMMARY:
**Summary:**

`_insert_value` function inserts a new key-value pair into `metadata` while handling the case when the value is `None`. Primary purpose: update a dictionary with a given name and value.

**Context:** Updating metadata with a specific key-value pair.

**Key details:**

* **Function signature:** `_insert_value(metadata, name, value)`
* **Implementation:** if `value` is `None`, returns original `metadata`; otherwise, updates `metadata[name] = value`
* **Inherited interfaces:** No explicit inheritance; relying on dictionary operations.


key: chroma:document
string_value:
CODE:

def is_hf_dataset(dataset):
    if not is_datasets_available():
        return False

    from datasets import Dataset, IterableDataset



SUMMARY:
**Summary**

This is a Python function `is_hf_dataset` that takes a `dataset` as input and returns a boolean indicating whether it's an HDF dataset.

### Key Details

* Checks if the necessary `datasets` library is available.
* Verifies if the provided `dataset` is either of type `Dataset` or `IterableDataset`.


key: chroma:document
string_value:
CODE:

def _get_mapping_values(mapping):
    result = []
    for v in mapping.values():
        if isinstance(v, (tuple, list)):
            result += list(v)
        else:
            result.append(v)


SUMMARY:
**Summary:**

`_get_mapping_values` function:

* Takes a dictionary `mapping` as input
* Flattens nested values (tuples or lists) into a single list
* Returns the flattened values in ascending order

**Key details:**

* Implemented using a simple for loop and type checks (`isinstance`)
* Uses the `+=` operator to prepend new elements to an existing list (`result += list(v)`)

**Notable dependencies/requirements:**

* None mentioned, but likely relies on built-in Python data structures and types.


key: chroma:document
string_value:
CODE:

def parse_keras_history(logs):
    """
    Parse the `logs` of either a `keras.History` object returned by `model.fit()` or an accumulated logs `dict`
    passed to the `PushToHubCallback`. Returns lines and logs compatible with those returned by `parse_log_history`.
    """
    if hasattr(logs, "history"):
        # This looks like a `History` object
        if not hasattr(logs, "epoch"):
            # This history looks empty, return empty results
            return None, [], {}
        logs.history["epoch"] = logs.epoch
        logs = logs.history
    else:
        # Training logs is a list of dicts, let's invert it to a dict of lists to match a History object
        logs = {log_key: [single_dict[log_key] for single_dict in logs] for log_key in logs[0]}

    lines = []
    for i in range(len(logs["epoch"])):
        epoch_dict = {log_key: log_value_list[i] for log_key, log_value_list in logs.items()}
        values = {}
        for k, v in epoch_dict.items():
            if k.startswith("val_"):
                k = "validation_" + k[4:]
            elif k != "epoch":
                k = "train_" + k
            splits = k.split("_")
            name = " ".join([part.capitalize() for part in splits])
            values[name] = v
        lines.append(values)

    eval_results = lines[-1]



SUMMARY:
**Summary:**

### Function Purpose and Functionality

`parse_keras_history(logs)` parses the Keras training history object or an accumulated log dictionary into a structured format.

**Main Purpose:** Convert Keras history data to match `parse_log_history` output, allowing for easier analysis and comparison of model performance metrics across different epochs.

### Key Implementation Details

*   Handles both instances of `keras.History` objects and accumulated log dictionaries as input.
*   Inverts the log dictionary into a format similar to `keras.History` if it's not already in that form.
*   Iterates over each epoch, normalizing metric keys (e.g., "val_loss" becomes "validation_loss") and extracting relevant values.

**Notable Dependencies**

*   Keras library
*   ModelFit()
*   PushToHubCallback

### Notable Code Sections

*   Conditionals for handling `History` objects vs. log dictionaries:
    ```python
if hasattr(logs, "history"):
    # Handle History object
else:
    # Invert log dictionary to match History format
```


key: chroma:document
string_value:
CODE:

def parse_log_history(log_history):
    """
    Parse the `log_history` of a Trainer to get the intermediate and final evaluation results.
    """
    idx = 0
    while idx < len(log_history) and "train_runtime" not in log_history[idx]:
        idx += 1

    # If there are no training logs
    if idx == len(log_history):
        idx -= 1
        while idx >= 0 and "eval_loss" not in log_history[idx]:
            idx -= 1

        if idx >= 0:
            return None, None, log_history[idx]
        else:
            return None, None, None

    # From now one we can assume we have training logs:
    train_log = log_history[idx]
    lines = []
    training_loss = "No log"
    for i in range(idx):
        if "loss" in log_history[i]:
            training_loss = log_history[i]["loss"]
        if "eval_loss" in log_history[i]:
            metrics = log_history[i].copy()
            _ = metrics.pop("total_flos", None)
            epoch = metrics.pop("epoch", None)
            step = metrics.pop("step", None)
            _ = metrics.pop("eval_runtime", None)
            _ = metrics.pop("eval_samples_per_second", None)
            _ = metrics.pop("eval_steps_per_second", None)
            _ = metrics.pop("eval_jit_compilation_time", None)
            values = {"Training Loss": training_loss, "Epoch": epoch, "Step": step}
            for k, v in metrics.items():
                if k == "eval_loss":
                    values["Validation Loss"] = v
                else:
                    splits = k.split("_")
                    name = " ".join([part.capitalize() for part in splits[1:]])
                    values[name] = v
            lines.append(values)

    idx = len(log_history) - 1
    while idx >= 0 and "eval_loss" not in log_history[idx]:
        idx -= 1

    if idx > 0:
        eval_results = {}
        for key, value in log_history[idx].items():
            if key.startswith("eval_"):
                key = key[5:]
            if key not in ["runtime", "samples_per_second", "steps_per_second", "epoch", "step"]:
                camel_cased_key = " ".join([part.capitalize() for part in key.split("_")])
                eval_results[camel_cased_key] = value
        return train_log, lines, eval_results
    else:


SUMMARY:
**Summary**

`parse_log_history` function:

* Parses log history to extract intermediate and final evaluation results for a `Trainer`.
* Returns tuple containing training log, epoch-wise metrics lines, and final evaluation metrics.

**Key Implementation Details**

* Iterates over log history to find first occurrence of "train_runtime" and then extracts relevant lines.
* Extracts training loss from earlier logs and populates metrics dictionary with required key-value pairs.
* Creates list of epoch-wise metrics lines by iterating over intermediate logs and appending values.
* Iterates over final logs to extract evaluation metrics (excluding "runtime", "samples_per_second", etc.).
* Returns training log, epoch-wise metrics lines, and final evaluation metrics.

**Notable Dependencies or Requirements**

* `log_history`: Input log history string or list of strings to parse.
* Uses:
	+ `metrics` dictionary to store key-value pairs for epoch-wise metrics lines.
	+ `_` (implicit assignment) to remove unnecessary values from dictionaries.


key: chroma:document
string_value:
CODE:

def extract_hyperparameters_from_keras(model):
    from .modeling_tf_utils import keras

    hyperparameters = {}
    if hasattr(model, "optimizer") and model.optimizer is not None:
        hyperparameters["optimizer"] = model.optimizer.get_config()
    else:
        hyperparameters["optimizer"] = None
    hyperparameters["training_precision"] = keras.mixed_precision.global_policy().name



SUMMARY:
**Function Summary: `extract_hyperparameters_from_keras`**

Extracts hyperparameter values from a Keras model.

**Key Details:**

*   Retrieves optimizer settings, including configuration and global precision policy.
*   Utilizes `keras.mixed_precision.global_policy().name` to determine training precision setting.

**Method Signatures and Interfaces:**

*   Returns a dictionary (`hyperparameters`) containing extracted hyperparameter values.

**Dependent Functions/Classes:**

*   `.modeling_tf_utils.keras`
*   Keras model's optimizer (conditionally)

**External Dependencies:**

*   Keras library (imported as `keras` from `.modeling_tf_utils`)
*   TensorFlow Mixed Precision Policy library


key: chroma:document
string_value:
CODE:

def _maybe_round(v, decimals=4):
    if isinstance(v, float) and len(str(v).split(".")) > 1 and len(str(v).split(".")[1]) > decimals:
        return f"{v:.{decimals}f}"


SUMMARY:
**Summary:**

`_maybe_round` function formats a float value to specify decimal places, rounding if necessary.

* Purpose:
	+ Rounds or truncates floating-point numbers within the specified decimal place range.
	+ Defaults to 4 decimal places.
* Implementation Details:
	+ Uses conditional statement with `isinstance()`, string manipulation (`str(v).split(".")`), and f-string formatting (`f"{v:.{decimals}f}"`).
* Notable Dependencies/Requirements: None.

**Key Method Signatures and Interfaces:** 
- `formatting(f_string, name, num_decimal_places)`

**External Dependencies:** N/A

**Inherited or Implemented Interfaces:** None.


key: chroma:document
string_value:
CODE:

def _regular_table_line(values, col_widths):
    values_with_space = [f"| {v}" + " " * (w - len(v) + 1) for v, w in zip(values, col_widths)]


SUMMARY:
**Summary:**

`_regular_table_line` function formats table row data with aligned columns.

**Key Details:**

* Takes two parameters: `values` (table content) and `col_widths` (column widths)
* Uses string formatting to add whitespace and create aligned columns
* Joins formatted values with a newline character

**Implementation Notable:**

* Utilizes the `"%"` alignment format specifer to align columns

**Dependencies/Requirements:**

* None, but assumes use of Python's built-in `zip`, `f-string`, and string formatting functionality.


key: chroma:document
string_value:
CODE:

def _second_table_line(col_widths):
    values = ["|:" + "-" * w + ":" for w in col_widths]


SUMMARY:
**Summary:**

`_second_table_line` function:

* Takes `col_widths` as input, where each width represents a table column
* Generates a string representation of the line separating two tables by iterating over `col_widths`, concatenating colon-separated dashes with pipes (`|:`)
* Returns the generated separator string along with a newline character (`\n`) at the end

**Key Implementation Details:**

* Primary function signature: `def _second_table_line(col_widths) -> str`
* Utilizes list comprehension to generate the separator string efficiently
* Leverages the `join` method to concatenate the values, eliminating unnecessary concatenation operations
* Includes a newline character (`\n`) at the end of the returned string


key: chroma:document
string_value:
CODE:

def make_markdown_table(lines):
    """
    Create a nice Markdown table from the results in `lines`.
    """
    if lines is None or len(lines) == 0:
        return ""
    col_widths = {key: len(str(key)) for key in lines[0].keys()}
    for line in lines:
        for key, value in line.items():
            if col_widths[key] < len(_maybe_round(value)):
                col_widths[key] = len(_maybe_round(value))

    table = _regular_table_line(list(lines[0].keys()), list(col_widths.values()))
    table += _second_table_line(list(col_widths.values()))
    for line in lines:
        table += _regular_table_line([_maybe_round(v) for v in line.values()], list(col_widths.values()))


SUMMARY:
**Concise Summary**

The `make_markdown_table` function creates a Markdown table from the provided input data, represented as a dictionary of lines. It calculates column widths based on row values and generates a complete table, including header and separator lines.

**Details:**

* **Functionality:** Calculates column widths for each key in the input data and constructs a Markdown table.
* **Key Algorithm/Pattern:** Uses dictionaries to store and manipulate data, with iterative processing of each line item.

**Important Method Signatures and Interfaces:**

* `_regular_table_line()`: Takes lists of headers and values as arguments
* `_second_table_line()`: Takes list of column widths as an argument
* `_maybe_round()`: Returns rounded value ( exact implementation not shown)

**Notable Dependencies or Requirements:**

* External dependency: None identified
* Inherited/Implemented interfaces: Assuming existence of `_regular_table_line`, `_second_table_line` and `_maybe_round` functions/functions with defined interfaces.


key: chroma:document
string_value:
CODE:

def extract_hyperparameters_from_trainer(trainer):
    hyperparameters = {k: getattr(trainer.args, k) for k in _TRAINING_ARGS_KEYS}

    if trainer.args.parallel_mode not in [ParallelMode.NOT_PARALLEL, ParallelMode.NOT_DISTRIBUTED]:
        hyperparameters["distributed_type"] = (
            "multi-GPU" if trainer.args.parallel_mode == ParallelMode.DISTRIBUTED else trainer.args.parallel_mode.value
        )
    if trainer.args.world_size > 1:
        hyperparameters["num_devices"] = trainer.args.world_size
    if trainer.args.gradient_accumulation_steps > 1:
        hyperparameters["gradient_accumulation_steps"] = trainer.args.gradient_accumulation_steps

    total_train_batch_size = (
        trainer.args.train_batch_size * trainer.args.world_size * trainer.args.gradient_accumulation_steps
    )
    if total_train_batch_size != hyperparameters["train_batch_size"]:
        hyperparameters["total_train_batch_size"] = total_train_batch_size
    total_eval_batch_size = trainer.args.eval_batch_size * trainer.args.world_size
    if total_eval_batch_size != hyperparameters["eval_batch_size"]:
        hyperparameters["total_eval_batch_size"] = total_eval_batch_size

    if trainer.args.optim:
        optimizer_name = trainer.args.optim
        optimizer_args = trainer.args.optim_args if trainer.args.optim_args else "No additional optimizer arguments"

        if "adam" in optimizer_name.lower():
            hyperparameters["optimizer"] = (
                f"Use {optimizer_name} with betas=({trainer.args.adam_beta1},{trainer.args.adam_beta2}) and"
                f" epsilon={trainer.args.adam_epsilon} and optimizer_args={optimizer_args}"
            )
        else:
            hyperparameters["optimizer"] = f"Use {optimizer_name} and the args are:\n{optimizer_args}"

    hyperparameters["lr_scheduler_type"] = trainer.args.lr_scheduler_type.value
    if trainer.args.warmup_ratio != 0.0:
        hyperparameters["lr_scheduler_warmup_ratio"] = trainer.args.warmup_ratio
    if trainer.args.warmup_steps != 0.0:
        hyperparameters["lr_scheduler_warmup_steps"] = trainer.args.warmup_steps
    if trainer.args.max_steps != -1:
        hyperparameters["training_steps"] = trainer.args.max_steps
    else:
        hyperparameters["num_epochs"] = trainer.args.num_train_epochs

    if trainer.args.fp16:
        if trainer.use_apex:
            hyperparameters["mixed_precision_training"] = f"Apex, opt level {trainer.args.fp16_opt_level}"
        else:
            hyperparameters["mixed_precision_training"] = "Native AMP"

    if trainer.args.label_smoothing_factor != 0.0:
        hyperparameters["label_smoothing_factor"] = trainer.args.label_smoothing_factor



SUMMARY:
**Function Summary:**

`extract_hyperparameters_from_trainer`

**Purpose and Functionality:**

This function extracts key hyperparameters from a `Trainer` object, including distributed training settings, optimizer configurations, learning rate schedulers, and mixed precision training options.

**Key Implementation Details:**

* Extracts hyperparameter values using `getattr` for each key in `_TRAINING_ARGS_KEYS`
* Computes total train and eval batch sizes based on `world_size`, `gradient_accumulation_steps`, and other training settings
* Processes optimizer configurations, including AdamW and mixed precision training options
* Updates hyperparameters with learning rate scheduler settings and label smoothing factor

**Notable Dependencies or Requirements:**

* `Trainer` object with attributes for training settings (e.g., `_TRAINING_ARGS_KEYS`, `parallel_mode`, `world_size`)
* `_TRAINING_ARGS_KEYS` list defining valid hyperparameter keys
* `ParallelMode` enum class for parallel mode determination

**Interfaces:**

* Uses ` getattr` to extract attribute values from the `Trainer` object's namespace.

This summary preserves exact class/function variable names and provides a concise overview of the function's purpose, key implementation details, and dependencies or requirements.


key: chroma:document
string_value:
CODE:

class ChannelDimension(ExplicitEnum):
    FIRST = "channels_first"


SUMMARY:
**Class Summary**

`ChannelDimension` is an `ExplicitEnum` that defines two possible dimension configurations for channels.

**Key Implementation Details**
- `FIRST`: Channels are presented in the order of their most recently added dimensions at the beginning (`channels_first`).
- `LAST`: Channels are presented in the reverse order of their most recently added dimensions at the end (`channels_last`).

**Notable Dependencies or Requirements**

- Inherits from `ExplicitEnum`, a base class in Python standard library.

No external dependencies mentioned.


key: chroma:document
string_value:
CODE:

class AnnotationFormat(ExplicitEnum):
    COCO_DETECTION = "coco_detection"


SUMMARY:
**Summary**

`AnnotationFormat` is an `ExplicitEnum` class defining two annotation formats: `COCO_DETECTION` and `COCO_PANOPTIC`. These values are used to specify the format of annotations in object detection models, such as COCO (Common Objects in Context) detection. No external dependencies or notable implementation details are present.


key: chroma:document
string_value:
CODE:

class AnnotionFormat(ExplicitEnum):
    COCO_DETECTION = AnnotationFormat.COCO_DETECTION.value


SUMMARY:
**Summary:**

This code defines an enumeration `AnnotionFormat` with two values, `COCO_DETECTION` and `COCO_PANOPTIC`, which mirror the corresponding constants in the `AnnotationFormat` class.

**Key Details:**

* **Implementation:** The values are assigned using the `.value` attribute of the parent `AnnotationFormat` enum.
* **Primary Purpose:** To provide a readable and maintainable way to represent different annotation formats.
* **Notable Interfaces:** Inherited from `ExplicitEnum`, which typically provides additional utility methods for enum-like behavior.

**Dependences:**

* `AnnotationFormat` class (parent enum)


key: chroma:document
string_value:
CODE:

def is_pil_image(img):


SUMMARY:
**Summary:**

`is_pil_image` function checks if an image object `img` is a valid Pillow Image object. It returns `True` if both `vision` functionality is available and the input `img` is an instance of PIL's `Image.Image`.

**Key Implementation Details:**

- Checks availability of `vision` functionality using `is_vision_available()` 
- Verifies image type using `isinstance(img, PIL.Image.Image)`

**External Dependencies:**

- Requires Pillow library (`PIL`)


key: chroma:document
string_value:
CODE:

class ImageType(ExplicitEnum):
    PIL = "pillow"
    TORCH = "torch"
    NUMPY = "numpy"
    TENSORFLOW = "tensorflow"


SUMMARY:
**Summary:**

The `ImageType` class ( ExplicitEnum ) defines a set of enumeration values representing different image processing technologies:

*   `PIL`, mapped to "pillow"
*   `TORCH`, mapped to "torch"
*   `NUMPY`, mapped to "numpy"
*   `TENSORFLOW`, mapped to "tensorflow"
*   `JAX`, mapped to "jax"

This class likely serves as a way to restrict or formalize image processing technology choices within an application or library.


key: chroma:document
string_value:
CODE:

def get_image_type(image):
    if is_pil_image(image):
        return ImageType.PIL
    if is_torch_tensor(image):
        return ImageType.TORCH
    if is_numpy_array(image):
        return ImageType.NUMPY
    if is_tf_tensor(image):
        return ImageType.TENSORFLOW
    if is_jax_tensor(image):
        return ImageType.JAX


SUMMARY:
**Function Summary:**

`get_image_type(image)` 
=========================

Determines the type of an input image based on its underlying format.

*   Purpose:
    *   Identifies the image type, specifically its technology stack (PIL, Torch, NumPy, TensorFlow, JAX).
    *   Raises a ValueError if the image type is unrecognized.

**Key Implementation Details:**

*   Uses multiple conditional checks to determine the image type.
*   Utilizes external functions:
    *   `is_pil_image(image)`: Checks if the image is in the Python Imaging Library (PIL) format.
    *   `is_torch_tensor(image)`, `is_numpy_array(image)`, `is_tf_tensor(image)`, and `is_jax_tensor(image)`: Check for respective formats in Torch, NumPy, TensorFlow, and JAX.

**Notable Dependencies:**

*   External dependencies include PIL, Torch (Torch library), NumPy, TensorFlow, and JAX.

Note that the function's exact type (i.e., Python function or class method) is not specified due to an lack of context; however, it is likely a utility function in a larger framework such as image processing.


key: chroma:document
string_value:
CODE:

def is_valid_image(img):


SUMMARY:
**Summary:**

```python
def is_valid_image(img):
```

*   **Purpose:** Determines if an input image is valid by checking its type against multiple formats using separate functions (`is_pil_image`, `is_numpy_array`, `is_torch_tensor`, `is_tf_tensor`, and `is_jax_tensor`).
*   **Key Implementation Details:**
    *   Uses multiple checks for different image formats.
    *   Returns a boolean value indicating validation status.

**Related Imports/Classes/Functions:**

No explicit references, but it's assumed that the functions (`is_pil_image`, `is_numpy_array`, `is_torch_tensor`, `is_tf_tensor`, and `is_jax_tensor`) are defined elsewhere in the codebase.
```python
import PIL
import numpy as np
import torch
import tensorflow as tf
import jax
```
Note: The exact import statements might be different depending on the actual implementation.


key: chroma:document
string_value:
CODE:

def valid_images(imgs):
    # If we have an list of images, make sure every image is valid
    if isinstance(imgs, (list, tuple)):
        for img in imgs:
            if not valid_images(img):
                return False
    # If not a list of tuple, we have been given a single image or batched tensor of images
    elif not is_valid_image(imgs):
        return False


SUMMARY:
**Summary:**

The `valid_images` function checks whether one or more images (or batches of images) are valid. It recursively verifies each image in the input list, returning `False` if any image is invalid. The primary purpose is to validate image integrity.

**Key Analysis Points:**

* Algorithm: Recursive image validation using list/tuple traversal
* Design Choice: Separate recursive function for handling nested lists/ranges
* Important Method Signatures:
	+ `valid_images(img)` (base case)
	+ `is_valid_image(imgs)` (used to verify single/batch images)

**Notable Dependencies:**

* External dependency on an implicit 'image validation' function (`is_valid_image`)
* Inherent assumptions about the image data structure and its validity checks


key: chroma:document
string_value:
CODE:

def is_batched(img):
    if isinstance(img, (list, tuple)):
        return is_valid_image(img[0])


SUMMARY:
**Function Summary: `is_batched`**

The `is_batched` function checks if an image input (`img`) is batched. It performs the following operations:

* If the input is a list or tuple, it calls `is_valid_image` on its first element (`img[0]`).
* Otherwise, it returns `False`.

**Key Implementation Details:**

* Uses `isinstance` to check the type of the input.
* Calls another function `is_valid_image` with the image data.

**Notable Dependencies/Requirements:**

* `is_valid_image` function (not shown in this code snippet).

This summary provides a concise overview of the function's purpose and key implementation details, without including unnecessary information.


key: chroma:document
string_value:
CODE:

def is_scaled_image(image: np.ndarray) -> bool:
    """
    Checks to see whether the pixel values have already been rescaled to [0, 1].
    """
    if image.dtype == np.uint8:
        return False

    # It's possible the image has pixel values in [0, 255] but is of floating type


SUMMARY:
**Summary**

```markdown
# Function: is_scaled_image
## Purpose
Checks if an image has pixel values in the [0, 1] range.

### Description

Returns `True` if the pixel values are already rescaled to `[0, 1]`. Checks for the following conditions:
- The image uses **unsigned 8-bit integer** data type (`np.uint8`), which implies scaling is not applied.
- The minimum and maximum pixel values are within the scaled range `[0, 1]`.

### Implementation

```python
def is_scaled_image(image: np.ndarray) -> bool:
    # Checks if image uses unsigned 8-bit integer data type and pixel values are in [0, 255]
    # Assume floating-point representation is present if min and max are outside [0, 1] range.
```

**Key Insights**

- The function assumes a `np.uint8` data type unless the pixel values indicate another representation.
- It relies on `np.min(image)` and `np.max(image)` being greater than or equal to 0 and less than or equal to 1 for floating-point images.


key: chroma:document
string_value:
CODE:

def make_list_of_images(images, expected_ndims: int = 3) -> List[ImageInput]:
    """
    Ensure that the input is a list of images. If the input is a single image, it is converted to a list of length 1.
    If the input is a batch of images, it is converted to a list of images.

    Args:
        images (`ImageInput`):
            Image of images to turn into a list of images.
        expected_ndims (`int`, *optional*, defaults to 3):
            Expected number of dimensions for a single input image. If the input image has a different number of
            dimensions, an error is raised.
    """
    if is_batched(images):
        return images

    # Either the input is a single image, in which case we create a list of length 1
    if isinstance(images, PIL.Image.Image):
        # PIL images are never batched
        return [images]

    if is_valid_image(images):
        if images.ndim == expected_ndims + 1:
            # Batch of images
            images = list(images)
        elif images.ndim == expected_ndims:
            # Single image
            images = [images]
        else:
            raise ValueError(
                f"Invalid image shape. Expected either {expected_ndims + 1} or {expected_ndims} dimensions, but got"
                f" {images.ndim} dimensions."
            )
        return images
    raise ValueError(
        "Invalid image type. Expected either PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or "
        f"jax.ndarray, but got {type(images)}."


SUMMARY:
**Function Summary: `make_list_of_images`**

Converts input images to a list of images while ensuring the expected number of dimensions.

**Purpose and Functionality**
This function takes an image or batch of images as input, checks its type and dimensionality, and returns a list of images. If the input is a single image, it is converted to a list with one element. If the input is already a list of images, no conversion is performed.

**Key Implementation Details**

* Uses `PIL Image` class for single-image inputs
* Verifies expected number of dimensions (3 by default) using `ndim` attribute
* Converts batch of images or single image to list using indexing or slicing

**Notable Dependencies and Requirements**
- PIL (Python Imaging Library)
- Numpy
- PyTorch/TensorFlow/JuliaNN (for valid image types)

Please note that exact method signatures, interfaces, and class names are preserved in the summary for accuracy.


key: chroma:document
string_value:
CODE:

def to_numpy_array(img) -> np.ndarray:
    if not is_valid_image(img):
        raise ValueError(f"Invalid image type: {type(img)}")

    if is_vision_available() and isinstance(img, PIL.Image.Image):
        return np.array(img)


SUMMARY:
**Summary**

A function `tonumpy_array` that converts an input image to a NumPy array:
- Validates the image type using `is_valid_image`.
  * Relying on explicit comparison `type(img)`.
  - Checks if vision is available and if the image is of type PIL.Image.Image.
    * Import of PIL library seems implied, assume `PIL` in scope.
  - If valid and compatible, returns the array conversion using NumPy's `array`.
  - Otherwise, falls back to `to_numpy` method.

**Key Details**

* Method return type: `np.ndarray`
* Interface checked for `vision` availability
* Image validation: `is_valid_image`

**Notable Dependency**
* `PIL.Image.Image`: implied import


key: chroma:document
string_value:
CODE:

def infer_channel_dimension_format(
    image: np.ndarray, num_channels: Optional[Union[int, Tuple[int, ...]]] = None
) -> ChannelDimension:
    """
    Infers the channel dimension format of `image`.

    Args:
        image (`np.ndarray`):
            The image to infer the channel dimension of.
        num_channels (`int` or `Tuple[int, ...]`, *optional*, defaults to `(1, 3)`):
            The number of channels of the image.

    Returns:
        The channel dimension of the image.
    """
    num_channels = num_channels if num_channels is not None else (1, 3)
    num_channels = (num_channels,) if isinstance(num_channels, int) else num_channels

    if image.ndim == 3:
        first_dim, last_dim = 0, 2
    elif image.ndim == 4:
        first_dim, last_dim = 1, 3
    else:
        raise ValueError(f"Unsupported number of image dimensions: {image.ndim}")

    if image.shape[first_dim] in num_channels and image.shape[last_dim] in num_channels:
        logger.warning(
            f"The channel dimension is ambiguous. Got image shape {image.shape}. Assuming channels are the first dimension."
        )
        return ChannelDimension.FIRST
    elif image.shape[first_dim] in num_channels:
        return ChannelDimension.FIRST
    elif image.shape[last_dim] in num_channels:
        return ChannelDimension.LAST


SUMMARY:
**Summary**

This function, `infer_channel_dimension_format`, determines the channel dimension format of an input image. The main purpose is to classify the channel dimension as either `.FIRST` or `.LAST`.

**Key Implementation Details**

* Checks if the image has 3 or 4 dimensions and adjusts the dimension indices accordingly.
* Verifies if the first/last dimension's size is present in `num_channels`. Falls back to default values if ambiguous.

**Notable Dependencies/Requirements**

* None

**Returns:**

* `ChannelDimension.FIRST` or `ChannelDimension.LAST`, indicating whether the channel dimension is in the first or last position, respectively.


key: chroma:document
string_value:
CODE:

def get_channel_dimension_axis(
    image: np.ndarray, input_data_format: Optional[Union[ChannelDimension, str]] = None
) -> int:
    """
    Returns the channel dimension axis of the image.

    Args:
        image (`np.ndarray`):
            The image to get the channel dimension axis of.
        input_data_format (`ChannelDimension` or `str`, *optional*):
            The channel dimension format of the image. If `None`, will infer the channel dimension from the image.

    Returns:
        The channel dimension axis of the image.
    """
    if input_data_format is None:
        input_data_format = infer_channel_dimension_format(image)
    if input_data_format == ChannelDimension.FIRST:
        return image.ndim - 3
    elif input_data_format == ChannelDimension.LAST:
        return image.ndim - 1


SUMMARY:
**Function Summary**

`get_channel_dimension_axis(image, input_data_format=None) -> int`

Retrieves the channel dimension axis of an image based on its `input_data_format`.

**Key Details**

* Returns `image.ndim - 3` if `input_data_format` is `ChannelDimension.FIRST`
* Returns `image(ndim - 1)` if `input_data_format` is `ChannelDimension.LAST`

**Dependencies/Requirements**

* `infer_channel_dimension_format(image)`
* `ChannelDimension`
* Supports channels in dimensions: `(2, 128`, ...,), (3, 256, ...)

**Notes on Implementation**

Uses a simple conditional statement to handle supported data formats. Raises an error for unsupported formats.


key: chroma:document
string_value:
CODE:

def get_image_size(image: np.ndarray, channel_dim: ChannelDimension = None) -> Tuple[int, int]:
    """
    Returns the (height, width) dimensions of the image.

    Args:
        image (`np.ndarray`):
            The image to get the dimensions of.
        channel_dim (`ChannelDimension`, *optional*):
            Which dimension the channel dimension is in. If `None`, will infer the channel dimension from the image.

    Returns:
        A tuple of the image's height and width.
    """
    if channel_dim is None:
        channel_dim = infer_channel_dimension_format(image)

    if channel_dim == ChannelDimension.FIRST:
        return image.shape[-2], image.shape[-1]
    elif channel_dim == ChannelDimension.LAST:
        return image.shape[-3], image.shape[-2]
    else:


SUMMARY:
**Summary**

`get_image_size` function returns the dimensions of an input image. It takes two parameters:

*   `image`: A 3D numpy array representing the image.
*   `channel_dim`: The dimension where the channel dimension is located (optional; defaults to inference).

The function returns a tuple containing the height and width of the image.

**Implementation Details**

*   Inferences channel dimension if provided.
*   Returns dimensions using `image.shape[-2]` and `image.shape[-1]` for `ChannelDimension.FIRST` and `ChannelDimension.LAST`, respectively.

**Dependencies/Requirements**

*   `infer_channel_dimension_format`: A function to infer the channel dimension format from an image (not shown in this code snippet).
*   `ChannelDimension`: An enum representing the possible data formats (FIRST and LAST).

**Method Signatures and Interfaces**

*   `image.shape[-2]` and `image.shape[-1]`.

Note that the actual implementation of `infer_channel_dimension_format` is not provided, as it is assumed to be a separate function. Additionally, the `ChannelDimension` enum is also not defined in this code snippet.


key: chroma:document
string_value:
CODE:

def is_valid_annotation_coco_detection(annotation: Dict[str, Union[List, Tuple]]) -> bool:
    if (
        isinstance(annotation, dict)
        and "image_id" in annotation
        and "annotations" in annotation
        and isinstance(annotation["annotations"], (list, tuple))
        and (
            # an image can have no annotations
            len(annotation["annotations"]) == 0 or isinstance(annotation["annotations"][0], dict)
        )
    ):
        return True


SUMMARY:
**Summary: `is_valid_annotation_coco_detection` Function**

This function determines whether a given annotation is valid for COCO Detection format. It checks if the input `annotation` adheres to the following structure:
- Has an "image_id" key
- Has an "annotations" key containing a list or tuple of annotations, with at least one being a dictionary

**Implementation Details:**

* Uses `isinstance()` to check type and validate nested structures.
* Returns `True` if the annotation is valid and falsy otherwise.

**Notable Dependencies:**

* None explicitly listed
* May rely on the `collections` library ( implicit: `Dict`, `Union`, `List`, `Tuple`)


key: chroma:document
string_value:
CODE:

def is_valid_annotation_coco_panoptic(annotation: Dict[str, Union[List, Tuple]]) -> bool:
    if (
        isinstance(annotation, dict)
        and "image_id" in annotation
        and "segments_info" in annotation
        and "file_name" in annotation
        and isinstance(annotation["segments_info"], (list, tuple))
        and (
            # an image can have no segments
            len(annotation["segments_info"]) == 0 or isinstance(annotation["segments_info"][0], dict)
        )
    ):
        return True


SUMMARY:
**Summary: `is_valid_annotation_coco_panoptic` Function**

This function checks if an annotation dictionary is valid for the COCO panoptic segmentation format. It verifies that the input annotation contains required keys ("image_id", "segments_info", and "file_name") and ensures that "segments_info" is either a list or tuple containing segments with correct structure.

**Key implementation details:**

* Uses `isinstance` checks to verify types of annotations
* Enforces strict structure for valid "segments_info"
* Returns `True` if annotation is valid, `False` otherwise

**Notable dependencies and requirements:**

* `Dict[str, Union[List, Tuple]]`: Input annotation type hint
* `Len(annotation["segments_info"]) == 0 or isinstance(annotation["segments_info"][0], dict)`: Condition for valid segments format
* No external dependencies listed


key: chroma:document
string_value:
CODE:

def valid_coco_detection_annotations(annotations: Iterable[Dict[str, Union[List, Tuple]]]) -> bool:


SUMMARY:
**Summary:**

```python
def valid_coco_detection_annotations(annotations: Iterable[Dict[str, Union[List, Tuple]]]) -> bool:
    # Returns True if all COCO detection annotations are valid, False otherwise

    def is_valid_annotation_coco_detection(ann: Dict[str, Union[List, Tuple]]) -> bool:
        # Internal function to validate a single annotation
        pass  # implementation not shown in this snippet
```

**Key Analysis Points:**

- **Primary Purpose:** Checks if all COCO detection annotations are valid.
- **Implementation Focus:** The function iterates over the annotated objects using an iterable and calls `is_valid_annotation_coco_detection` on each one, returning a boolean result based on all values being True.


key: chroma:document
string_value:
CODE:

def valid_coco_panoptic_annotations(annotations: Iterable[Dict[str, Union[List, Tuple]]]) -> bool:


SUMMARY:
**Summary**

This function, `valid_coco_panoptic_annotations`, checks the validity of COCO Panoptic annotations. It ensures that all annotations in the input iterable conform to the annotation format using the `is_valid_annotation_coco_panoptic` helper function. The function returns `True` if all annotations are valid.

**Key Implementation Details**

* The function uses the `all` function with a generator expression to iterate over annotated objects (`ann`) in the input iterable.
* It relies on the `is_valid_annotation_coco_panoptic` function, which is not shown in this code snippet, to determine whether each annotation is valid.
* The function returns the result of the `all` iteration.

**Notable Dependencies or Requirements**

* A function or class named `is_valid_annotation_coco_panoptic`, whose implementation is not provided.


key: chroma:document
string_value:
CODE:

def load_image(image: Union[str, "PIL.Image.Image"], timeout: Optional[float] = None) -> "PIL.Image.Image":
    """
    Loads `image` to a PIL Image.

    Args:
        image (`str` or `PIL.Image.Image`):
            The image to convert to the PIL Image format.
        timeout (`float`, *optional*):
            The timeout value in seconds for the URL request.

    Returns:
        `PIL.Image.Image`: A PIL Image.
    """
    requires_backends(load_image, ["vision"])
    if isinstance(image, str):
        if image.startswith("http://") or image.startswith("https://"):
            # We need to actually check for a real protocol, otherwise it's impossible to use a local file
            # like http_huggingface_co.png
            image = PIL.Image.open(BytesIO(requests.get(image, timeout=timeout).content))
        elif os.path.isfile(image):
            image = PIL.Image.open(image)
        else:
            if image.startswith("data:image/"):
                image = image.split(",")[1]

            # Try to load as base64
            try:
                b64 = base64.decodebytes(image.encode())
                image = PIL.Image.open(BytesIO(b64))
            except Exception as e:
                raise ValueError(
                    f"Incorrect image source. Must be a valid URL starting with `http://` or `https://`, a valid path to an image file, or a base64 encoded string. Got {image}. Failed with {e}"
                )
    elif isinstance(image, PIL.Image.Image):
        image = image
    else:
        raise TypeError(
            "Incorrect format used for image. Should be an url linking to an image, a base64 string, a local path, or a PIL image."
        )
    image = PIL.ImageOps.exif_transpose(image)
    image = image.convert("RGB")


SUMMARY:
**Summary:**

### Function Purpose and Functionality
`load_image`: A function to load an image to a PIL Image format, supporting various input formats - string (URL, base64), path, or PIL Image. Loads the requested image with a given timeout.

### Key Implementation Details
- **Algorithm:** Converts input image from various sources to PIL Image, applying necessary operations such as URL request, decoding base64, and handling local files.
- **Design Choice:** Uses `PIL.ImageOps.exif_transpose` and then `convet("RGB")` for processing the loaded images.

### Notable Dependencies or Requirements
- `requests` (external dependency) for URL requests with timeout
- `PIL` (Python Imaging Library) (required dependency)
- `base64`, `os`, and `BytesIO` (built-in Python modules)

**Method Signatures and Interfaces:**
```python
load_image(str, Optional[float]) -> PIL.Image.Image
```

Note: The RAG system will store this summary for reference.


key: chroma:document
string_value:
CODE:

def load_images(
    images: Union[List, Tuple, str, "PIL.Image.Image"], timeout: Optional[float] = None
) -> Union["PIL.Image.Image", List["PIL.Image.Image"], List[List["PIL.Image.Image"]]]:
    """Loads images, handling different levels of nesting.

    Args:
      images: A single image, a list of images, or a list of lists of images to load.
      timeout: Timeout for loading images.

    Returns:
      A single image, a list of images, a list of lists of images.
    """
    if isinstance(images, (list, tuple)):
        if len(images) and isinstance(images[0], (list, tuple)):
            return [[load_image(image, timeout=timeout) for image in image_group] for image_group in images]
        else:
            return [load_image(image, timeout=timeout) for image in images]
    else:


SUMMARY:
**Function Summary:**

`load_images` loads one or more images from various nested configurations and returns the loaded results.

```
Parameters:
  - `images`: An image (singleton), list of images, or list of lists of images to load.
  - Optional `timeout`

Returns:
  A single image instance, list of images, or list of lists of images
```

**Key Implementation Details:**

* Recursively handles nested configurations using `if isinstance(images, (list, tuple))`
* Uses list comprehension to process different types of input data (`images` and `image_groups`)
* Calls `load_image` recursively to handle each image

**Notable Dependencies/Interfaces:**

* Import from PIL library (`"PIL.Image"` instances)
* Type checking for function arguments and return values
* Optional argument for loading under a timeout


key: chroma:document
string_value:
CODE:

def validate_preprocess_arguments(
    do_rescale: Optional[bool] = None,
    rescale_factor: Optional[float] = None,
    do_normalize: Optional[bool] = None,
    image_mean: Optional[Union[float, List[float]]] = None,
    image_std: Optional[Union[float, List[float]]] = None,
    do_pad: Optional[bool] = None,
    size_divisibility: Optional[int] = None,
    do_center_crop: Optional[bool] = None,
    crop_size: Optional[Dict[str, int]] = None,
    do_resize: Optional[bool] = None,
    size: Optional[Dict[str, int]] = None,
    resample: Optional["PILImageResampling"] = None,
):
    """
    Checks validity of typically used arguments in an `ImageProcessor` `preprocess` method.
    Raises `ValueError` if arguments incompatibility is caught.
    Many incompatibilities are model-specific. `do_pad` sometimes needs `size_divisor`,
    sometimes `size_divisibility`, and sometimes `size`. New models and processors added should follow
    existing arguments when possible.

    """
    if do_rescale and rescale_factor is None:
        raise ValueError("`rescale_factor` must be specified if `do_rescale` is `True`.")

    if do_pad and size_divisibility is None:
        # Here, size_divisor might be passed as the value of size
        raise ValueError(
            "Depending on the model, `size_divisibility`, `size_divisor`, `pad_size` or `size` must be specified if `do_pad` is `True`."
        )

    if do_normalize and (image_mean is None or image_std is None):
        raise ValueError("`image_mean` and `image_std` must both be specified if `do_normalize` is `True`.")

    if do_center_crop and crop_size is None:
        raise ValueError("`crop_size` must be specified if `do_center_crop` is `True`.")

    if do_resize and (size is None or resample is None):


SUMMARY:
**Summary:**

Validate Preprocess Arguments Function:

This function validates the input arguments for an `ImageProcessor` `preprocess` method.

**Purpose:** Validate preprocessing argument incompatibilities, raising `ValueError` if any are found.

**Key Implementation Details:**

* Checks if required parameters are specified when others are set (e.g., `do_rescale` and `/rescale_factor`, `/do_pad` and `/size_divisibility`, etc.).
* Raises `ValueError` for incompatibilities, noting model-specific exceptions.

**Notable Parameters:**

* `/do_rescale`: Rescales input image.
* `/rescale_factor`: Scaling factor for rescaling (optional).
* `/do_normalize`: Normalizes pixel values by adding zero mean and scaling to unit variance.
* `/image_mean` and `/image_std`: Used for normalization if `/do_normalize` is True.
* `/do_pad`: Pads output image with zeros.
* `/size_divisibility`: Size divisibility requirement when pads are specified (optional).
* `/do_center_crop`: Crops center region of input image.
* `/crop_size`: Crop size requirements when center cropping is enabled (optional).
* `/do_resize`: Resizes output image to specified size.
* `/size` and `/resample`: Resize output image if `/do_resize` is True.

**External Dependencies:**

* None explicitly listed, but may depend on PIL library for `PILImageResampling`.


key: chroma:document
string_value:
CODE:

def validate_fast_preprocess_arguments(
    do_rescale: Optional[bool] = None,
    rescale_factor: Optional[float] = None,
    do_normalize: Optional[bool] = None,
    image_mean: Optional[Union[float, List[float]]] = None,
    image_std: Optional[Union[float, List[float]]] = None,
    do_pad: Optional[bool] = None,
    size_divisibility: Optional[int] = None,
    do_center_crop: Optional[bool] = None,
    crop_size: Optional[Dict[str, int]] = None,
    do_resize: Optional[bool] = None,
    size: Optional[Dict[str, int]] = None,
    resample: Optional["PILImageResampling"] = None,
    return_tensors: Optional[Union[str, TensorType]] = None,
    data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
):
    """
    Checks validity of typically used arguments in an `ImageProcessorFast` `preprocess` method.
    Raises `ValueError` if arguments incompatibility is caught.
    """
    validate_preprocess_arguments(
        do_rescale=do_rescale,
        rescale_factor=rescale_factor,
        do_normalize=do_normalize,
        image_mean=image_mean,
        image_std=image_std,
        do_resize=do_resize,
        size=size,
        resample=resample,
    )
    # Extra checks for ImageProcessorFast
    if return_tensors != "pt":
        raise ValueError("Only returning PyTorch tensors is currently supported.")

    if data_format != ChannelDimension.FIRST:


SUMMARY:
**Summary:**

`validate_fast_preprocess_arguments` function validates typical arguments for the `ImageProcessorFast` preprocess method, also known as MobileNet Face Detector. It checks consistency of input parameters and raises `ValueError` if incompatibility is detected.

**Key Implementation Details:**

- Calls `validate_preprocess_statistics` with validation parameters.
- Checks that `return_tensors` equals `"pt"` to support only PyTorch tensors.
- Verifies that `data_format` equals `ChannelDimension.FIRST` for channel-first data format compatibility.

**Notable Dependencies/Requirements:**

* Uses `PILImageResampling`
* Expects `validate_preprocess_arguments`, a separate function not defined in this snippet
* Assumes `TensorType`, `Optional`, and `Union` type hints are correctly implemented.


key: chroma:document
string_value:
CODE:
# In the future we can add a TF implementation here when we have TF models.
class ImageFeatureExtractionMixin:
    """
    Mixin that contain utilities for preparing image features.
    """

    def _ensure_format_supported(self, image):
        if not isinstance(image, (PIL.Image.Image, np.ndarray)) and not is_torch_tensor(image):
            raise ValueError(
                f"Got type {type(image)} which is not supported, only `PIL.Image.Image`, `np.array` and "
                "`torch.Tensor` are."
            )

    def to_pil_image(self, image, rescale=None):
        """
        Converts `image` to a PIL Image. Optionally rescales it and puts the channel dimension back as the last axis if
        needed.

        Args:
            image (`PIL.Image.Image` or `numpy.ndarray` or `torch.Tensor`):
                The image to convert to the PIL Image format.
            rescale (`bool`, *optional*):
                Whether or not to apply the scaling factor (to make pixel values integers between 0 and 255). Will
                default to `True` if the image type is a floating type, `False` otherwise.
        """
        self._ensure_format_supported(image)

        if is_torch_tensor(image):
            image = image.numpy()

        if isinstance(image, np.ndarray):
            if rescale is None:
                # rescale default to the array being of floating type.
                rescale = isinstance(image.flat[0], np.floating)
            # If the channel as been moved to first dim, we put it back at the end.
            if image.ndim == 3 and image.shape[0] in [1, 3]:
                image = image.transpose(1, 2, 0)
            if rescale:
                image = image * 255
            image = image.astype(np.uint8)
            return PIL.Image.fromarray(image)
        return image

    def convert_rgb(self, image):
        """
        Converts `PIL.Image.Image` to RGB format.

        Args:
            image (`PIL.Image.Image`):
                The image to convert.
        """
        self._ensure_format_supported(image)
        if not isinstance(image, PIL.Image.Image):
            return image

        return image.convert("RGB")

    def rescale(self, image: np.ndarray, scale: Union[float, int]) -> np.ndarray:
        """
        Rescale a numpy image by scale amount
        """
        self._ensure_format_supported(image)
        return image * scale

    def to_numpy_array(self, image, rescale=None, channel_first=True):
        """
        Converts `image` to a numpy array. Optionally rescales it and puts the channel dimension as the first
        dimension.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to convert to a NumPy array.
            rescale (`bool`, *optional*):
                Whether or not to apply the scaling factor (to make pixel values floats between 0. and 1.). Will
                default to `True` if the image is a PIL Image or an array/tensor of integers, `False` otherwise.
            channel_first (`bool`, *optional*, defaults to `True`):
                Whether or not to permute the dimensions of the image to put the channel dimension first.
        """
        self._ensure_format_supported(image)

        if isinstance(image, PIL.Image.Image):
            image = np.array(image)

        if is_torch_tensor(image):
            image = image.numpy()

        rescale = isinstance(image.flat[0], np.integer) if rescale is None else rescale

        if rescale:
            image = self.rescale(image.astype(np.float32), 1 / 255.0)

        if channel_first and image.ndim == 3:
            image = image.transpose(2, 0, 1)

        return image

    def expand_dims(self, image):
        """
        Expands 2-dimensional `image` to 3 dimensions.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to expand.
        """
        self._ensure_format_supported(image)

        # Do nothing if PIL image
        if isinstance(image, PIL.Image.Image):
            return image

        if is_torch_tensor(image):
            image = image.unsqueeze(0)
        else:
            image = np.expand_dims(image, axis=0)
        return image

    def normalize(self, image, mean, std, rescale=False):
        """
        Normalizes `image` with `mean` and `std`. Note that this will trigger a conversion of `image` to a NumPy array
        if it's a PIL Image.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to normalize.
            mean (`List[float]` or `np.ndarray` or `torch.Tensor`):
                The mean (per channel) to use for normalization.
            std (`List[float]` or `np.ndarray` or `torch.Tensor`):
                The standard deviation (per channel) to use for normalization.
            rescale (`bool`, *optional*, defaults to `False`):
                Whether or not to rescale the image to be between 0 and 1. If a PIL image is provided, scaling will
                happen automatically.
        """
        self._ensure_format_supported(image)

        if isinstance(image, PIL.Image.Image):
            image = self.to_numpy_array(image, rescale=True)
        # If the input image is a PIL image, it automatically gets rescaled. If it's another
        # type it may need rescaling.
        elif rescale:
            if isinstance(image, np.ndarray):
                image = self.rescale(image.astype(np.float32), 1 / 255.0)
            elif is_torch_tensor(image):
                image = self.rescale(image.float(), 1 / 255.0)

        if isinstance(image, np.ndarray):
            if not isinstance(mean, np.ndarray):
                mean = np.array(mean).astype(image.dtype)
            if not isinstance(std, np.ndarray):
                std = np.array(std).astype(image.dtype)
        elif is_torch_tensor(image):
            import torch

            if not isinstance(mean, torch.Tensor):
                if isinstance(mean, np.ndarray):
                    mean = torch.from_numpy(mean)
                else:
                    mean = torch.tensor(mean)
            if not isinstance(std, torch.Tensor):
                if isinstance(std, np.ndarray):
                    std = torch.from_numpy(std)
                else:
                    std = torch.tensor(std)

        if image.ndim == 3 and image.shape[0] in [1, 3]:
            return (image - mean[:, None, None]) / std[:, None, None]
        else:
            return (image - mean) / std

    def resize(self, image, size, resample=None, default_to_square=True, max_size=None):
        """
        Resizes `image`. Enforces conversion of input to PIL.Image.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to resize.
            size (`int` or `Tuple[int, int]`):
                The size to use for resizing the image. If `size` is a sequence like (h, w), output size will be
                matched to this.

                If `size` is an int and `default_to_square` is `True`, then image will be resized to (size, size). If
                `size` is an int and `default_to_square` is `False`, then smaller edge of the image will be matched to
                this number. i.e, if height > width, then image will be rescaled to (size * height / width, size).
            resample (`int`, *optional*, defaults to `PILImageResampling.BILINEAR`):
                The filter to user for resampling.
            default_to_square (`bool`, *optional*, defaults to `True`):
                How to convert `size` when it is a single int. If set to `True`, the `size` will be converted to a
                square (`size`,`size`). If set to `False`, will replicate
                [`torchvision.transforms.Resize`](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Resize)
                with support for resizing only the smallest edge and providing an optional `max_size`.
            max_size (`int`, *optional*, defaults to `None`):
                The maximum allowed for the longer edge of the resized image: if the longer edge of the image is
                greater than `max_size` after being resized according to `size`, then the image is resized again so
                that the longer edge is equal to `max_size`. As a result, `size` might be overruled, i.e the smaller
                edge may be shorter than `size`. Only used if `default_to_square` is `False`.

        Returns:
            image: A resized `PIL.Image.Image`.
        """
        resample = resample if resample is not None else PILImageResampling.BILINEAR

        self._ensure_format_supported(image)

        if not isinstance(image, PIL.Image.Image):
            image = self.to_pil_image(image)

        if isinstance(size, list):
            size = tuple(size)

        if isinstance(size, int) or len(size) == 1:
            if default_to_square:
                size = (size, size) if isinstance(size, int) else (size[0], size[0])
            else:
                width, height = image.size
                # specified size only for the smallest edge
                short, long = (width, height) if width <= height else (height, width)
                requested_new_short = size if isinstance(size, int) else size[0]

                if short == requested_new_short:
                    return image

                new_short, new_long = requested_new_short, int(requested_new_short * long / short)

                if max_size is not None:
                    if max_size <= requested_new_short:
                        raise ValueError(
                            f"max_size = {max_size} must be strictly greater than the requested "
                            f"size for the smaller edge size = {size}"
                        )
                    if new_long > max_size:
                        new_short, new_long = int(max_size * new_short / new_long), max_size

                size = (new_short, new_long) if width <= height else (new_long, new_short)

        return image.resize(size, resample=resample)

    def center_crop(self, image, size):
        """
        Crops `image` to the given size using a center crop. Note that if the image is too small to be cropped to the
        size given, it will be padded (so the returned result has the size asked).

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor` of shape (n_channels, height, width) or (height, width, n_channels)):
                The image to resize.
            size (`int` or `Tuple[int, int]`):
                The size to which crop the image.

        Returns:
            new_image: A center cropped `PIL.Image.Image` or `np.ndarray` or `torch.Tensor` of shape: (n_channels,
            height, width).
        """
        self._ensure_format_supported(image)

        if not isinstance(size, tuple):
            size = (size, size)

        # PIL Image.size is (width, height) but NumPy array and torch Tensors have (height, width)
        if is_torch_tensor(image) or isinstance(image, np.ndarray):
            if image.ndim == 2:
                image = self.expand_dims(image)
            image_shape = image.shape[1:] if image.shape[0] in [1, 3] else image.shape[:2]
        else:
            image_shape = (image.size[1], image.size[0])

        top = (image_shape[0] - size[0]) // 2
        bottom = top + size[0]  # In case size is odd, (image_shape[0] + size[0]) // 2 won't give the proper result.
        left = (image_shape[1] - size[1]) // 2
        right = left + size[1]  # In case size is odd, (image_shape[1] + size[1]) // 2 won't give the proper result.

        # For PIL Images we have a method to crop directly.
        if isinstance(image, PIL.Image.Image):
            return image.crop((left, top, right, bottom))

        # Check if image is in (n_channels, height, width) or (height, width, n_channels) format
        channel_first = True if image.shape[0] in [1, 3] else False

        # Transpose (height, width, n_channels) format images
        if not channel_first:
            if isinstance(image, np.ndarray):
                image = image.transpose(2, 0, 1)
            if is_torch_tensor(image):
                image = image.permute(2, 0, 1)

        # Check if cropped area is within image boundaries
        if top >= 0 and bottom <= image_shape[0] and left >= 0 and right <= image_shape[1]:
            return image[..., top:bottom, left:right]

        # Otherwise, we may need to pad if the image is too small. Oh joy...
        new_shape = image.shape[:-2] + (max(size[0], image_shape[0]), max(size[1], image_shape[1]))
        if isinstance(image, np.ndarray):
            new_image = np.zeros_like(image, shape=new_shape)
        elif is_torch_tensor(image):
            new_image = image.new_zeros(new_shape)

        top_pad = (new_shape[-2] - image_shape[0]) // 2
        bottom_pad = top_pad + image_shape[0]
        left_pad = (new_shape[-1] - image_shape[1]) // 2
        right_pad = left_pad + image_shape[1]
        new_image[..., top_pad:bottom_pad, left_pad:right_pad] = image

        top += top_pad
        bottom += top_pad
        left += left_pad
        right += left_pad

        new_image = new_image[
            ..., max(0, top) : min(new_image.shape[-2], bottom), max(0, left) : min(new_image.shape[-1], right)
        ]

        return new_image

    def flip_channel_order(self, image):
        """
        Flips the channel order of `image` from RGB to BGR, or vice versa. Note that this will trigger a conversion of
        `image` to a NumPy array if it's a PIL Image.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image whose color channels to flip. If `np.ndarray` or `torch.Tensor`, the channel dimension should
                be first.
        """
        self._ensure_format_supported(image)

        if isinstance(image, PIL.Image.Image):
            image = self.to_numpy_array(image)

        return image[::-1, :, :]

    def rotate(self, image, angle, resample=None, expand=0, center=None, translate=None, fillcolor=None):
        """
        Returns a rotated copy of `image`. This method returns a copy of `image`, rotated the given number of degrees
        counter clockwise around its centre.

        Args:
            image (`PIL.Image.Image` or `np.ndarray` or `torch.Tensor`):
                The image to rotate. If `np.ndarray` or `torch.Tensor`, will be converted to `PIL.Image.Image` before
                rotating.

        Returns:
            image: A rotated `PIL.Image.Image`.
        """
        resample = resample if resample is not None else PIL.Image.NEAREST

        self._ensure_format_supported(image)

        if not isinstance(image, PIL.Image.Image):
            image = self.to_pil_image(image)

        return image.rotate(
            angle, resample=resample, expand=expand, center=center, translate=translate, fillcolor=fillcolor


SUMMARY:
It is an object-oriented class in Python for image processing. The main functionalities include resizing, flipping channel order, and rotating images.

Here are some example use cases:

**Resize**
```python
from PIL import Image

img = Image.open('image.jpg')
resized_img = obj.resize((800, 600))
```

**Flip Channel Order**
```python
from PIL import Image

img = Image.open('image.jpg')
flipped_img = obj.flip_channel_order(img)
```

**Rotate**
```python
from PIL import Image

img = Image.open('image.jpg')
rotated_img = obj.rotate(90, resample=PIL.Image.NEAREST)
```

The class appears to be well-structured and readable, with clear docstrings for each method. It also uses the Pillow library, which is a popular choice for image processing in Python.

However, there are a few suggestions that can improve the code:

* Some of the variable names could be more descriptive. For example, `top`, `bottom`, `left`, and `right` in the `flip_channel_order` method could be renamed to something like `height_offset` and `width_offset`.
* The class does not seem to have any methods for applying transformations to PIL Images. This could be added as a separate class with its own docstrings.
* Some of the constants, such as `PIL.Image.NEAREST`, could be moved outside the class or made into properties if they are used in multiple places.

Overall, the code is well-written and easy to understand. With some minor improvements, it could be even more readable and maintainable.


key: chroma:document
string_value:
CODE:

def validate_annotations(
    annotation_format: AnnotationFormat,
    supported_annotation_formats: Tuple[AnnotationFormat, ...],
    annotations: List[Dict],
) -> None:
    if annotation_format not in supported_annotation_formats:
        raise ValueError(f"Unsupported annotation format: {format} must be one of {supported_annotation_formats}")

    if annotation_format is AnnotationFormat.COCO_DETECTION:
        if not valid_coco_detection_annotations(annotations):
            raise ValueError(
                "Invalid COCO detection annotations. Annotations must a dict (single image) or list of dicts "
                "(batch of images) with the following keys: `image_id` and `annotations`, with the latter "
                "being a list of annotations in the COCO format."
            )

    if annotation_format is AnnotationFormat.COCO_PANOPTIC:
        if not valid_coco_panoptic_annotations(annotations):
            raise ValueError(
                "Invalid COCO panoptic annotations. Annotations must a dict (single image) or list of dicts "
                "(batch of images) with the following keys: `image_id`, `file_name` and `segments_info`, with "
                "the latter being a list of annotations in the COCO format."


SUMMARY:
**Function Summary**

`validate_annotations`: Validates input annotations against supported formats, raising errors for incompatible or malformed data.

```markdown
### Purpose and Functionality

* Validates `annotation_format` against `supported_annotation_formats`.
	+ Checks for supported format: `AnnotationFormat.annotation_format in supported_annotation_formats`
* Performs format-specific validation:
	+ COCO Detection: Validates `annotations` as either a dictionary (single image) or list of dictionaries (batch of images) with required keys.
	+ COCO Panoptic: Validates `annotations` as a dictionary (single image) or list of dictionaries with additional panoptic annotations.

### Implementation Details

* Uses functions:
	+ `valid_coco_detection_annotations`
	+ `valid_coco_panoptic_annotations`
* Raises errors for invalid formats, missing required keys, or incorrect annotation structure.
```

**Notable Dependencies**

* External dependencies: None
* Inherited interfaces:
	+ `AnnotationFormat`: supported in `supported_annotation_formats` tuple
	+ `Dict`, `List`, `Tuple`: used as return types and parameter types

**Key Method Signatures and Interfaces**

* `validate_annotations(annotation_format, supported_annotation_formats, annotations) -> None`
	+ Validates input annotations against supported formats.
* `valid_coco_detection_annotations(annotations) -> bool`
	+ Validates COCO detection annotations.
* `valid_coco_panoptic_annotations(annotations) -> bool`
	+ Validates COCO panoptic annotations.


key: chroma:document
string_value:
CODE:

def validate_kwargs(valid_processor_keys: List[str], captured_kwargs: List[str]):
    unused_keys = set(captured_kwargs).difference(set(valid_processor_keys))
    if unused_keys:
        unused_key_str = ", ".join(unused_keys)
        # TODO raise a warning here instead of simply logging?


SUMMARY:
**Function Summary**

`validate_kwargs` validates argument keywords against a predefined set of valid processor keys. It checks for unused or unrecognized keyword arguments and logs warnings.

**Key Implementation Details**

* Checks for and removes unused `captured_kwargs` not present in `valid_processor_keys`.
* Logs a warning when an unused key is found, using the joined string of all unused keys.
* Current implementation does not raise an exception; instead, it logs a warning.

**Notable Dependencies/Requirements**

* `List[str]`: The typesafe type hint for the input parameters.
* `logger` (assuming a logging module, e.g., Python's built-in `logging`): Used to log warnings about unused or unrecognized keywords.


key: chroma:document
string_value:
CODE:

class BaseImageProcessorFast(BaseImageProcessor):
    _transform_params = None

    def _build_transforms(self, **kwargs) -> "Compose":
        """
        Given the input settings e.g. do_resize, build the image transforms.
        """
        raise NotImplementedError

    def _validate_params(self, **kwargs) -> None:
        for k, v in kwargs.items():
            if k not in self._transform_params:
                raise ValueError(f"Invalid transform parameter {k}={v}.")

    @functools.lru_cache(maxsize=1)
    def get_transforms(self, **kwargs) -> "Compose":
        self._validate_params(**kwargs)
        return self._build_transforms(**kwargs)

    def to_dict(self):
        encoder_dict = super().to_dict()
        encoder_dict.pop("_transform_params", None)


SUMMARY:
**Summary**

The `BaseImageProcessorFast` class extends `BaseImageProcessor`. It serves as a base class for fast image processors, implementing the following key functionalities:

*   Validates input parameters against predefined transforms (_validate_params)
*   Builds image transforms based on input settings (_build_transforms)
*   Caches transform builds using memoization (get_transforms)
*   Converts an object to a dictionary (to_dict)

**Notable Details**

*   The `BaseImageProcessorFast` class reuses `_transform_params`, which is initialized as empty.
*   It inherits from `_build_transforms` method, leaving it unimplemented, indicating that it should be overridden by subclasses for specific transforms.
*   The `get_transforms` method relies on memoization to cache the results of `_build_transforms`.
*   The `to_dict` method removes an unsupported field (`_transform_params`) from the object's dictionary representation.

**Dependencies and Interfaces**

*   Extends `BaseImageProcessor`
*   Relies on private methods in `BaseImageProcessor`
*   None


key: chroma:document
string_value:
CODE:

def get_image_size_for_max_height_width(
    image_size: Tuple[int, int],
    max_height: int,
    max_width: int,
) -> Tuple[int, int]:
    """
    Computes the output image size given the input image and the maximum allowed height and width. Keep aspect ratio.
    Important, even if image_height < max_height and image_width < max_width, the image will be resized
    to at least one of the edges be equal to max_height or max_width.

    For example:
        - input_size: (100, 200), max_height: 50, max_width: 50 -> output_size: (25, 50)
        - input_size: (100, 200), max_height: 200, max_width: 500 -> output_size: (200, 400)

    Args:
        image_size (`Tuple[int, int]`):
            The image to resize.
        max_height (`int`):
            The maximum allowed height.
        max_width (`int`):
            The maximum allowed width.
    """
    height, width = image_size
    height_scale = max_height / height
    width_scale = max_width / width
    min_scale = min(height_scale, width_scale)
    new_height = int(height * min_scale)
    new_width = int(width * min_scale)


SUMMARY:
**Summary:**

`get_image_size_for_max_height_width` function calculates the output image size while maintaining the aspect ratio. Given input `image_size`, `max_height`, and `max_width`, it returns the smallest scaled dimensions to fit within the specified limits.

**Key Implementation Details:**

* Calculates scaling factors for each dimension (`height_scale`, `width_scale`)
* Computes the minimum scale factor (`min_scale`) that satisfies both constraints
* Applies this scale factor to calculate new dimensions (`new_height`, `new_width`)

**Notable Dependencies or Requirements:**

* Expects input as a tuple of two integers (image size)
* Inherently scales image based on aspect ratio preservation.


key: chroma:document
string_value:
CODE:

def safe_squeeze(tensor: "torch.Tensor", axis: Optional[int] = None) -> "torch.Tensor":
    """
    Squeezes a tensor, but only if the axis specified has dim 1.
    """
    if axis is None:
        return tensor.squeeze()

    try:
        return tensor.squeeze(axis=axis)
    except ValueError:


SUMMARY:
**Summary:**

`safe_squeeze` Function:

* Squeezes a PyTorch tensor along the specified axis (defaulting to all axes except 0 if no axis is provided).
* Exceptions are caught and swallowed by returning the original tensor, avoiding `ValueError`.

**Key Details:**

* `tensor.squeeze(axis=axis)` with defensive error handling.
* Axes 1-9 can be squeezed; axes 0 and other special axes not supported.

**Notable Dependencies:**

* PyTorch (`torch.Tensor`)
* Python's built-in `Optional` type for type hinting.


key: chroma:document
string_value:
CODE:

def max_across_indices(values: Iterable[Any]) -> List[Any]:
    """
    Return the maximum value across all indices of an iterable of values.
    """


SUMMARY:
**Summary**

```
**Functionality**: Returns the maximum value across all indices of an iterable of values.

**Implementation Details**:

* Uses `zip(*values)` to transpose and unpack the input iterable into separate lists for each index.
* Applies the `max` function on each of these lists using a list comprehension.

**Dependencies/Requirements**:
 None.
 
 Note: This summary only includes essential information about the code's functionality, implementation details, and dependencies. It adheres to RAG system guidelines by preserving exact class/function names, highlighting key algorithms (zip/unpacking), and omitting unnecessary details.


key: chroma:document
string_value:
CODE:

def get_max_height_width(images: List["torch.Tensor"]) -> Tuple[int]:
    """
    Get the maximum height and width across all images in a batch.
    """

    _, max_height, max_width = max_across_indices([img.shape for img in images])



SUMMARY:
**Function Summary: get_max_height_width**

Retrieves the maximum height and width from a batch of images using `torch.Tensor`.

### Implementation Details:

* Calculates maximum dimensions across image batches using `max_across_indices`.
* Returns a tuple containing the maximum height and width.

### Dependencies/Requirements:

* `torch`
* `List` and `Tuple` from Python's types module (`typing`)

### Context:

This function appears to be part of an image processing pipeline, suggesting it may be used in computer vision or machine learning applications.


key: chroma:document
string_value:
CODE:
# From https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse
def string_to_bool(v):
    if isinstance(v, bool):
        return v
    if v.lower() in ("yes", "true", "t", "y", "1"):
        return True
    elif v.lower() in ("no", "false", "f", "n", "0"):
        return False
    else:
        raise ArgumentTypeError(
            f"Truthy value expected: got {v} but expected one of yes/no, true/false, t/f, y/n, 1/0 (case insensitive)."


SUMMARY:
**Function Summary**

`string_to_bool`
================

Converts a string to its equivalent boolean value.

### Purpose
Parses a string into a boolean value, with optional case-insensitive conversion.

### Implementation Details

* Checks for explicit `bool` value
* Converts input strings to lowercase and checks for specified truthy values (yes/true/t/y/1) or falsey values (no/false/f/n/0)
* Raises `ArgumentTypeError` if invalid value is provided

### Notable Dependencies

* `isinstance`
* `str.lower`

### Method Signature
```
def string_to_bool(v: object) -> bool:
```


key: chroma:document
string_value:
CODE:

def make_choice_type_function(choices: list) -> Callable[[str], Any]:
    """
    Creates a mapping function from each choices string representation to the actual value. Used to support multiple
    value types for a single argument.

    Args:
        choices (list): List of choices.

    Returns:
        Callable[[str], Any]: Mapping function from string representation to actual value for each choice.
    """
    str_to_choice = {str(choice): choice for choice in choices}


SUMMARY:
**Summary:**

Creates a mapping function `make_choice_type_function` that maps string representations of choices to their actual values. This function uses a dictionary to store the mappings and returns a lambda function that looks up the choice value based on the input string representation.

**Key Implementation Details:**

* Maps each choice string to its corresponding value using a dictionary comprehension.
* Returns a lambda function with the signature `lambda arg: str_to_choice.get(arg, arg)`.

**Notable Dependencies or Requirements:**

* Uses the `Callable` type hint and the `Any` return type hint from the Python typing library.
* Inherit/Implements: No external interfaces are implemented. However, it uses the `list` type for the `choices` argument and returns a function with the `Callable[[str], Any]` signature.

**Context:** This code is likely part of a larger testing or validation framework to support multiple value types for a single argument.


key: chroma:document
string_value:
CODE:

def HfArg(
    *,
    aliases: Union[str, List[str]] = None,
    help: str = None,
    default: Any = dataclasses.MISSING,
    default_factory: Callable[[], Any] = dataclasses.MISSING,
    metadata: dict = None,
    **kwargs,
) -> dataclasses.Field:
    """Argument helper enabling a concise syntax to create dataclass fields for parsing with `HfArgumentParser`.

    Example comparing the use of `HfArg` and `dataclasses.field`:
    ```
    @dataclass
    class Args:
        regular_arg: str = dataclasses.field(default="Huggingface", metadata={"aliases": ["--example", "-e"], "help": "This syntax could be better!"})
        hf_arg: str = HfArg(default="Huggingface", aliases=["--example", "-e"], help="What a nice syntax!")
    ```

    Args:
        aliases (Union[str, List[str]], optional):
            Single string or list of strings of aliases to pass on to argparse, e.g. `aliases=["--example", "-e"]`.
            Defaults to None.
        help (str, optional): Help string to pass on to argparse that can be displayed with --help. Defaults to None.
        default (Any, optional):
            Default value for the argument. If not default or default_factory is specified, the argument is required.
            Defaults to dataclasses.MISSING.
        default_factory (Callable[[], Any], optional):
            The default_factory is a 0-argument function called to initialize a field's value. It is useful to provide
            default values for mutable types, e.g. lists: `default_factory=list`. Mutually exclusive with `default=`.
            Defaults to dataclasses.MISSING.
        metadata (dict, optional): Further metadata to pass on to `dataclasses.field`. Defaults to None.

    Returns:
        Field: A `dataclasses.Field` with the desired properties.
    """
    if metadata is None:
        # Important, don't use as default param in function signature because dict is mutable and shared across function calls
        metadata = {}
    if aliases is not None:
        metadata["aliases"] = aliases
    if help is not None:
        metadata["help"] = help



SUMMARY:
**Summary: `HfArg` Function**

The `HfArg` function provides a concise syntax to create dataclass fields for parsing with `HfArgumentParser`. It enables creating required or optional arguments with customizable aliases, help messages, and default values.

**Key Features:**

*   Creates a `dataclasses.Field` object with customizable properties:
    *   `aliases`: A list of string aliases to pass on to argparse.
    *   `help`: The help message to be displayed with `--help`.
    *   `default`: The default value for the argument.
    *   `default_factory`: A function called to initialize a field's value (mutually exclusive with `default`).
    *   Additional metadata (`**kwargs`).

**Method Signature:**
```python
def HfArg(
    *,
    aliases: Union[str, List[str]] = None,
    help: str = None,
    default: Any = dataclasses.MISSING,
    default_factory: Callable[[], Any] = dataclasses.Missing,
    metadata: dict = None,
    **kwargs,
) -> dataclasses.Field:
```

**Key Data Types and Functions Used:**

*   `Union[str, List[str]]`: Union type for aliases.
*   `Callable[[], Any]`: Function signature for `default_factory`.
*   `dataclasses.Field`: The class returned by `HfArg`.

This function provides a convenient way to create dataclass fields with customized properties, making it easier to parse arguments with `HfArgumentParser`.


key: chroma:document
string_value:
CODE:

class HfArgumentParser(ArgumentParser):
    """
    This subclass of `argparse.ArgumentParser` uses type hints on dataclasses to generate arguments.

    The class is designed to play well with the native argparse. In particular, you can add more (non-dataclass backed)
    arguments to the parser after initialization and you'll get the output back after parsing as an additional
    namespace. Optional: To create sub argument groups use the `_argument_group_name` attribute in the dataclass.
    """

    dataclass_types: Iterable[DataClassType]

    def __init__(self, dataclass_types: Union[DataClassType, Iterable[DataClassType]], **kwargs):
        """
        Args:
            dataclass_types:
                Dataclass type, or list of dataclass types for which we will "fill" instances with the parsed args.
            kwargs (`Dict[str, Any]`, *optional*):
                Passed to `argparse.ArgumentParser()` in the regular way.
        """
        # To make the default appear when using --help
        if "formatter_class" not in kwargs:
            kwargs["formatter_class"] = ArgumentDefaultsHelpFormatter
        super().__init__(**kwargs)
        if dataclasses.is_dataclass(dataclass_types):
            dataclass_types = [dataclass_types]
        self.dataclass_types = list(dataclass_types)
        for dtype in self.dataclass_types:
            self._add_dataclass_arguments(dtype)

    @staticmethod
    def _parse_dataclass_field(parser: ArgumentParser, field: dataclasses.Field):
        # Long-option strings are conventionlly separated by hyphens rather
        # than underscores, e.g., "--long-format" rather than "--long_format".
        # Argparse converts hyphens to underscores so that the destination
        # string is a valid attribute name. Hf_argparser should do the same.
        long_options = [f"--{field.name}"]
        if "_" in field.name:
            long_options.append(f"--{field.name.replace('_', '-')}")

        kwargs = field.metadata.copy()
        # field.metadata is not used at all by Data Classes,
        # it is provided as a third-party extension mechanism.
        if isinstance(field.type, str):
            raise RuntimeError(
                "Unresolved type detected, which should have been done with the help of "
                "`typing.get_type_hints` method by default"
            )

        aliases = kwargs.pop("aliases", [])
        if isinstance(aliases, str):
            aliases = [aliases]

        origin_type = getattr(field.type, "__origin__", field.type)
        if origin_type is Union or (hasattr(types, "UnionType") and isinstance(origin_type, types.UnionType)):
            if str not in field.type.__args__ and (
                len(field.type.__args__) != 2 or type(None) not in field.type.__args__
            ):
                raise ValueError(
                    "Only `Union[X, NoneType]` (i.e., `Optional[X]`) is allowed for `Union` because"
                    " the argument parser only supports one type per argument."
                    f" Problem encountered in field '{field.name}'."
                )
            if type(None) not in field.type.__args__:
                # filter `str` in Union
                field.type = field.type.__args__[0] if field.type.__args__[1] is str else field.type.__args__[1]
                origin_type = getattr(field.type, "__origin__", field.type)
            elif bool not in field.type.__args__:
                # filter `NoneType` in Union (except for `Union[bool, NoneType]`)
                field.type = (
                    field.type.__args__[0] if isinstance(None, field.type.__args__[1]) else field.type.__args__[1]
                )
                origin_type = getattr(field.type, "__origin__", field.type)

        # A variable to store kwargs for a boolean field, if needed
        # so that we can init a `no_*` complement argument (see below)
        bool_kwargs = {}
        if origin_type is Literal or (isinstance(field.type, type) and issubclass(field.type, Enum)):
            if origin_type is Literal:
                kwargs["choices"] = field.type.__args__
            else:
                kwargs["choices"] = [x.value for x in field.type]

            kwargs["type"] = make_choice_type_function(kwargs["choices"])

            if field.default is not dataclasses.MISSING:
                kwargs["default"] = field.default
            else:
                kwargs["required"] = True
        elif field.type is bool or field.type == Optional[bool]:
            # Copy the currect kwargs to use to instantiate a `no_*` complement argument below.
            # We do not initialize it here because the `no_*` alternative must be instantiated after the real argument
            bool_kwargs = copy(kwargs)

            # Hack because type=bool in argparse does not behave as we want.
            kwargs["type"] = string_to_bool
            if field.type is bool or (field.default is not None and field.default is not dataclasses.MISSING):
                # Default value is False if we have no default when of type bool.
                default = False if field.default is dataclasses.MISSING else field.default
                # This is the value that will get picked if we don't include --{field.name} in any way
                kwargs["default"] = default
                # This tells argparse we accept 0 or 1 value after --{field.name}
                kwargs["nargs"] = "?"
                # This is the value that will get picked if we do --{field.name} (without value)
                kwargs["const"] = True
        elif isclass(origin_type) and issubclass(origin_type, list):
            kwargs["type"] = field.type.__args__[0]
            kwargs["nargs"] = "+"
            if field.default_factory is not dataclasses.MISSING:
                kwargs["default"] = field.default_factory()
            elif field.default is dataclasses.MISSING:
                kwargs["required"] = True
        else:
            kwargs["type"] = field.type
            if field.default is not dataclasses.MISSING:
                kwargs["default"] = field.default
            elif field.default_factory is not dataclasses.MISSING:
                kwargs["default"] = field.default_factory()
            else:
                kwargs["required"] = True
        parser.add_argument(*long_options, *aliases, **kwargs)

        # Add a complement `no_*` argument for a boolean field AFTER the initial field has already been added.
        # Order is important for arguments with the same destination!
        # We use a copy of earlier kwargs because the original kwargs have changed a lot before reaching down
        # here and we do not need those changes/additional keys.
        if field.default is True and (field.type is bool or field.type == Optional[bool]):
            bool_kwargs["default"] = False
            parser.add_argument(
                f"--no_{field.name}",
                f"--no-{field.name.replace('_', '-')}",
                action="store_false",
                dest=field.name,
                **bool_kwargs,
            )

    def _add_dataclass_arguments(self, dtype: DataClassType):
        if hasattr(dtype, "_argument_group_name"):
            parser = self.add_argument_group(dtype._argument_group_name)
        else:
            parser = self

        try:
            type_hints: Dict[str, type] = get_type_hints(dtype)
        except NameError:
            raise RuntimeError(
                f"Type resolution failed for {dtype}. Try declaring the class in global scope or "
                "removing line of `from __future__ import annotations` which opts in Postponed "
                "Evaluation of Annotations (PEP 563)"
            )
        except TypeError as ex:
            # Remove this block when we drop Python 3.9 support
            if sys.version_info[:2] < (3, 10) and "unsupported operand type(s) for |" in str(ex):
                python_version = ".".join(map(str, sys.version_info[:3]))
                raise RuntimeError(
                    f"Type resolution failed for {dtype} on Python {python_version}. Try removing "
                    "line of `from __future__ import annotations` which opts in union types as "
                    "`X | Y` (PEP 604) via Postponed Evaluation of Annotations (PEP 563). To "
                    "support Python versions that lower than 3.10, you need to use "
                    "`typing.Union[X, Y]` instead of `X | Y` and `typing.Optional[X]` instead of "
                    "`X | None`."
                ) from ex
            raise

        for field in dataclasses.fields(dtype):
            if not field.init:
                continue
            field.type = type_hints[field.name]
            self._parse_dataclass_field(parser, field)

    def parse_args_into_dataclasses(
        self,
        args=None,
        return_remaining_strings=False,
        look_for_args_file=True,
        args_filename=None,
        args_file_flag=None,
    ) -> Tuple[DataClass, ...]:
        """
        Parse command-line args into instances of the specified dataclass types.

        This relies on argparse's `ArgumentParser.parse_known_args`. See the doc at:
        docs.python.org/3.7/library/argparse.html#argparse.ArgumentParser.parse_args

        Args:
            args:
                List of strings to parse. The default is taken from sys.argv. (same as argparse.ArgumentParser)
            return_remaining_strings:
                If true, also return a list of remaining argument strings.
            look_for_args_file:
                If true, will look for a ".args" file with the same base name as the entry point script for this
                process, and will append its potential content to the command line args.
            args_filename:
                If not None, will uses this file instead of the ".args" file specified in the previous argument.
            args_file_flag:
                If not None, will look for a file in the command-line args specified with this flag. The flag can be
                specified multiple times and precedence is determined by the order (last one wins).

        Returns:
            Tuple consisting of:

                - the dataclass instances in the same order as they were passed to the initializer.abspath
                - if applicable, an additional namespace for more (non-dataclass backed) arguments added to the parser
                  after initialization.
                - The potential list of remaining argument strings. (same as argparse.ArgumentParser.parse_known_args)
        """

        if args_file_flag or args_filename or (look_for_args_file and len(sys.argv)):
            args_files = []

            if args_filename:
                args_files.append(Path(args_filename))
            elif look_for_args_file and len(sys.argv):
                args_files.append(Path(sys.argv[0]).with_suffix(".args"))

            # args files specified via command line flag should overwrite default args files so we add them last
            if args_file_flag:
                # Create special parser just to extract the args_file_flag values
                args_file_parser = ArgumentParser()
                args_file_parser.add_argument(args_file_flag, type=str, action="append")

                # Use only remaining args for further parsing (remove the args_file_flag)
                cfg, args = args_file_parser.parse_known_args(args=args)
                cmd_args_file_paths = vars(cfg).get(args_file_flag.lstrip("-"), None)

                if cmd_args_file_paths:
                    args_files.extend([Path(p) for p in cmd_args_file_paths])

            file_args = []
            for args_file in args_files:
                if args_file.exists():
                    file_args += args_file.read_text().split()

            # in case of duplicate arguments the last one has precedence
            # args specified via the command line should overwrite args from files, so we add them last
            args = file_args + args if args is not None else file_args + sys.argv[1:]
        namespace, remaining_args = self.parse_known_args(args=args)
        outputs = []
        for dtype in self.dataclass_types:
            keys = {f.name for f in dataclasses.fields(dtype) if f.init}
            inputs = {k: v for k, v in vars(namespace).items() if k in keys}
            for k in keys:
                delattr(namespace, k)
            obj = dtype(**inputs)
            outputs.append(obj)
        if len(namespace.__dict__) > 0:
            # additional namespace.
            outputs.append(namespace)
        if return_remaining_strings:
            return (*outputs, remaining_args)
        else:
            if remaining_args:
                raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")

            return (*outputs,)

    def parse_dict(self, args: Dict[str, Any], allow_extra_keys: bool = False) -> Tuple[DataClass, ...]:
        """
        Alternative helper method that does not use `argparse` at all, instead uses a dict and populating the dataclass
        types.

        Args:
            args (`dict`):
                dict containing config values
            allow_extra_keys (`bool`, *optional*, defaults to `False`):
                Defaults to False. If False, will raise an exception if the dict contains keys that are not parsed.

        Returns:
            Tuple consisting of:

                - the dataclass instances in the same order as they were passed to the initializer.
        """
        unused_keys = set(args.keys())
        outputs = []
        for dtype in self.dataclass_types:
            keys = {f.name for f in dataclasses.fields(dtype) if f.init}
            inputs = {k: v for k, v in args.items() if k in keys}
            unused_keys.difference_update(inputs.keys())
            obj = dtype(**inputs)
            outputs.append(obj)
        if not allow_extra_keys and unused_keys:
            raise ValueError(f"Some keys are not used by the HfArgumentParser: {sorted(unused_keys)}")
        return tuple(outputs)

    def parse_json_file(
        self, json_file: Union[str, os.PathLike], allow_extra_keys: bool = False
    ) -> Tuple[DataClass, ...]:
        """
        Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the
        dataclass types.

        Args:
            json_file (`str` or `os.PathLike`):
                File name of the json file to parse
            allow_extra_keys (`bool`, *optional*, defaults to `False`):
                Defaults to False. If False, will raise an exception if the json file contains keys that are not
                parsed.

        Returns:
            Tuple consisting of:

                - the dataclass instances in the same order as they were passed to the initializer.
        """
        with open(Path(json_file), encoding="utf-8") as open_json_file:
            data = json.loads(open_json_file.read())
        outputs = self.parse_dict(data, allow_extra_keys=allow_extra_keys)
        return tuple(outputs)

    def parse_yaml_file(
        self, yaml_file: Union[str, os.PathLike], allow_extra_keys: bool = False
    ) -> Tuple[DataClass, ...]:
        """
        Alternative helper method that does not use `argparse` at all, instead loading a yaml file and populating the
        dataclass types.

        Args:
            yaml_file (`str` or `os.PathLike`):
                File name of the yaml file to parse
            allow_extra_keys (`bool`, *optional*, defaults to `False`):
                Defaults to False. If False, will raise an exception if the json file contains keys that are not
                parsed.

        Returns:
            Tuple consisting of:

                - the dataclass instances in the same order as they were passed to the initializer.
        """
        outputs = self.parse_dict(yaml.safe_load(Path(yaml_file).read_text()), allow_extra_keys=allow_extra_keys)


SUMMARY:
This is an implementation of a Python class called `HfArgumentParser` that serves as a parser for command-line arguments and configuration files in the context of Hugging Face's Hub. The class provides several methods for parsing different formats, including `argparse`, JSON, YAML, and a dictionary.

Here are some key features:

*   **Supports multiple input formats**: The `HfArgumentParser` supports parsing from `argparse` objects, as well as CSV and dictionaries.
*   **Configuration file support**: It can parse configuration files in JSON and YAML format.
*   **Dataclass support**: It provides a method to create instances of dataclasses using the parsed values.

Some potential modifications or improvements could be:

*   **Error handling**: Adding more specific error messages for different types of input errors would improve user experience.
*   **Config file validation**: While JSON and YAML files can contain extra keys, the parser currently allows them. Introducing optional schema validation to enforce strict adherence to expected dataclass attributes might enhance robustness.
*   **Type hints for method parameters**: Adding type hints could improve code readability by specifying the types of all input parameters.

This class appears to be designed as a utility within an NLP or machine learning workflow, possibly with tools for working on specific tasks like model training or inference.


key: chroma:document
string_value:
CODE:

class PytorchGELUTanh(nn.Module):
    """
    A fast C implementation of the tanh approximation of the GeLU activation function. See
    https://arxiv.org/abs/1606.08415.

    This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical
    match due to rounding errors.
    """

    def __init__(self):
        super().__init__()
        if version.parse(torch.__version__) < version.parse("1.12.0"):
            raise ImportError(
                f"You are using torch=={torch.__version__}, but torch>=1.12.0 is required to use "
                "PytorchGELUTanh. Please upgrade torch."
            )

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**PytorchGELUTanh Class Summary**

**Main Purpose and Functionality**
The PytorchGELUTanh class implements the tanh approximation of the GeLU activation function in PyTorch, providing a fast and optimized alternative to NewGELU and FastGELU.

**Key Implementation Details**

* Uses `nn.functional.gelu` with `approximate="tanh"` to simulate GeLU behavior similar to NewGELU and FastGELU.
* Rounding errors introduce non-exact numerical matching.

**Notable Dependencies and Requirements**

* Requires `torch>=1.12.0` for functionality; version less than 1.12.0 will raise an error.
* Uses `nn.functional.gelu` from PyTorch functional module.


key: chroma:document
string_value:
CODE:

class NewGELUActivation(nn.Module):
    """
    Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see
    the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415
    """

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**Summary:**

The `NewGELUActivation` class implements the GELU (Gaussian Error Linear Unit) activation function from the paper "Gaussian Error Linear Units" by Hendrycks and Gimpel (2016). The forward method takes an input tensor, applies element-wise operations to compute a scaled and shifted version of the input, which is then passed through a sigmoid function.

**Key Implementation Details:**

* Applies scaling factor `sqrt(2.0 / pi)` to input to stabilize gradients
* Employs cubing (`torch.pow(input, 3.0)`) for robustness improvement
* Incorporates sigmoid (`torch.tanh` with inverse hyperbolic sine calculation) to transition from linear and ReLU-like behavior

**Notable Dependencies:**

* `torch`: PyTorch library for tensor computations and activation functions
* `math`: Python standard library for mathematical operations (sqrt, pi)


key: chroma:document
string_value:
CODE:

class GELUActivation(nn.Module):
    """
    Original Implementation of the GELU activation function in Google BERT repo when initially created. For
    information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 +
    torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional
    Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415
    """

    def __init__(self, use_gelu_python: bool = False):
        super().__init__()
        if use_gelu_python:
            self.act = self._gelu_python
        else:
            self.act = nn.functional.gelu

    def _gelu_python(self, input: Tensor) -> Tensor:
        return input * 0.5 * (1.0 + torch.erf(input / math.sqrt(2.0)))

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**GELUActivation Class Summary**

**Purpose and Functionality:**
The `GELUActivation` class implements the Generalized Erifunctional Unit (GELU) activation function. It provides two different implementations: one written in Python (`_gelu_python`) for flexibility, and another using the native PyTorch implementation (`nn.functional.gelu`). The primary purpose is to apply the GELU activation function to input tensors.

**Key Implementation Details:**

* Uses `torch.erf` to compute the error function.
* Employs a hybrid approach with two implementable methods (_gelu_python and the native `nn.functional.gelu`).
 
**Notable Dependencies/Requirements:**
- `matplotlib` (not explicitly used but referenced in related math functions)
- `math` for mathematical operations like sqrt
- `torch.nn.functional` for native implementation


key: chroma:document
string_value:
CODE:

class FastGELUActivation(nn.Module):
    """
    Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs
    """

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**FastGELUActivation Class Summary**

### Purpose and Functionality
This class implements the FastGELU (Gaussian Error Linear Units) activation function, which approximates the standard GELU but is less computationally intensive. It is designed to provide accurate results.

### Key Implementation Details

* The `forward` method takes an input tensor and returns a tensor with the output of the GELU approximation.
* The formula used is: `0.5 * input * (1.0 + torch.tanh(input * 0.7978845608 * (1.0 + 0.044715 * input * input)))`

### Notable Dependencies and Requirements

* `torch`: a PyTorch module required for numerical computations.
* `nn.Module`: the base class for PyTorch neural network modules, inherited by this class.

### Interface
`fastgelu_activation(nn.Module)`

### Return Types

`Tensor`
```


key: chroma:document
string_value:
CODE:

class QuickGELUActivation(nn.Module):
    """
    Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs
    """

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**Summary**

The `QuickGELUActivation` class implements a fast but inaccurate GELU (Gaussian Error Linear Unit) activation function using the `torch` library.

* Primary purpose: Approximates the GELU function for faster computations.
* Key implementation details:
	+ Uses the Gaussian error linear unit function, approximately implemented as `x * sigmoid(1.702 * x)`.

**Code Analysis**

* The class inherits from `nn.Module`, indicating it is a PyTorch neural network module.
* The `forward` method takes an input tensor and returns the output of the approximation, utilizing the `torch` library for the sigmoid function compute.
* No external dependencies are listed in the code snippet. However, it relies on the `torch` library for its implementation.

**Notable Dependencies**

None explicitly listed; relies on `torch` as primary dependency.


key: chroma:document
string_value:
CODE:

class ClippedGELUActivation(nn.Module):
    """
    Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as
    it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to
    https://arxiv.org/abs/2004.09602.

    Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when
    initially created.

    For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 +
    torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://arxiv.org/abs/1606.08415
    """

    def __init__(self, min: float, max: float):
        if min > max:
            raise ValueError(f"min should be < max (got min: {min}, max: {max})")

        super().__init__()
        self.min = min
        self.max = max

    def forward(self, x: Tensor) -> Tensor:


SUMMARY:
**Class Summary: ClippedGELUActivation**

The `ClippedGELUActivation` class is a custom neural network module that applies the clipped GELU (Gaussian Error Linear Unit) activation function. It reduces the range of possible GELu outputs between `[min, max]` to make the activation function more suitable for quantization.

**Key Implementation Details:**

* The `forward` method takes input `x` and returns the clipped GE LU output using `torch.clip`.
* The `__init__` method initializes the module with `min` and `max` values, raising an error if `min` is greater than `max`.

**Notable Dependencies:**

* `gelu`: The original implementation of the GE LU activation function.
* `torch`: Used for mathematical operations.

**Interfaces Implemented:**

* `nn.Module`
* `TensorProcessing` (via `torch.clip`)
* `ActivationFunction` (with custom clipped implementation)


key: chroma:document
string_value:
CODE:

class AccurateGELUActivation(nn.Module):
    """
    Applies GELU approximation that is faster than default and more accurate than QuickGELU. See:
    https://github.com/hendrycks/GELUs

    Implemented along with MEGA (Moving Average Equipped Gated Attention)
    """

    def __init__(self):
        super().__init__()
        self.precomputed_constant = math.sqrt(2 / math.pi)

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**AccurateGELUActivation Class Summary**

**Main Purpose and Functionality:**

The `AccurateGELUActivation` class implements an Accurate GELU (Gaussian Error Linear Unit) activation function using the precomputed constant from Hendryck's implementation. It is optimized for speed and accuracy compared to default GELU and QuickGELU.

**Key Implementation Details:**

* The activation function is computed using `torch.tanh`, with a custom precomputed constant (`self.precomputed_constant`) to improve performance.
* The input tensor is scaled by a factor (`0.5`) before applying the GELU transformation.

**Notable Dependencies or Requirements:**

* `torch`: PyTorch library, specifically for computational operations such as `math` and `torch.tanh`.
* Relevant classes:
	+ `Tensor`: PyTorch's numerical data structure.
* External dependencies:
	+ Hendryck's GitHub implementation of GELU (`https://github.com/hendrycks/GELUs`).


key: chroma:document
string_value:
CODE:

class MishActivation(nn.Module):
    """
    See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also
    visit the official repository for the paper: https://github.com/digantamisra98/Mish
    """

    def __init__(self):
        super().__init__()
        if version.parse(torch.__version__) < version.parse("1.9.0"):
            self.act = self._mish_python
        else:
            self.act = nn.functional.mish

    def _mish_python(self, input: Tensor) -> Tensor:
        return input * torch.tanh(nn.functional.softplus(input))

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**Concise Summary: Mish Activation Function Implementation**

The `MishActivation` class implements the Mitra et al.'s Mish activation function in PyTorch. Its primary purpose is to provide a self-regularized, non-monotonic activation function.

**Key Features:**

* Conditionally uses either the implementation from the Python repository (`_mish_python`) or PyTorch's built-in `nn.functional.mish`.
* `_mish_python` function applies `torch.tanh` and `nn.functional.softplus` to a given input.

**Notable Dependencies:**

* PyTorch library (version-dependent)
* Official Mish repository (optional, used in Python implementation)

**Important Class/Function Signatures:**

* `forward(input: Tensor) -> Tensor`: Takes an input tensor and applies the activation function.
* `_mish_python(input: Tensor) -> Tensor`: Applies the Mish activation function using PyTorch's softplus and tanh for computation.


key: chroma:document
string_value:
CODE:

class LinearActivation(nn.Module):
    """
    Applies the linear activation function, i.e. forwarding input directly to output.
    """

    def forward(self, input: Tensor) -> Tensor:


SUMMARY:
**Class Summary**

`LinearActivation`

A neural network module that applies the linear activation function by passing the input tensor directly to the output.

### Key Implementation Details

- **Forward Pass**: The `forward` method takes an input tensor and returns it unchanged, effectively applying no transformation.
  - Returns: `Tensor`
  - Method Signature: `input (Tensor) -> Tensor`

### Notable Dependencies or Requirements

- **Tensor**: Input and output tensors using the PyTorch library (`torch.Tensor`).


key: chroma:document
string_value:
CODE:

class LaplaceActivation(nn.Module):
    """
    Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See
    https://arxiv.org/abs/2209.10655

    Inspired by squared relu, but with bounded range and gradient for better stability
    """

    def forward(self, input, mu=0.707107, sigma=0.282095):
        input = (input - mu).div(sigma * math.sqrt(2.0))


SUMMARY:
**LaplaceActivation Class Summary**

The `LaplaceActivation` class is an implementation of the Laplace activation function, introduced as an attention activation in the MEGA paper.

**Primary Purpose:**
The primary purpose of this class is to apply element-wise activation based on the Laplace function.

**Key Implementation Details:**

* The `forward` method calculates the input using the following formula:
	+ `[input - mu] / (sigma * sqrt(2.0))`: standardization around mean (`mu`) and standard deviation (`sigma`)
	+ `torch.erf(input)`: computes the error function of the standardized input
	+ `0.5 * (1.0 + torch.erf(input))`: applies activation to produce output

**Notable Dependencies:**

* Reliably referenced classes and functions:
	+ `torch.erf()`, `math.sqrt()` for computing error function and square root respectively.
* External dependencies:
	+ None explicitly listed, but it is assumed that the PyTorch library (`nn.Module`) is present.

Note this code appears to be incomplete in its implementation as there's no return statement declared outside of the return statement of `torch.erf(input)*0.5`


key: chroma:document
string_value:
CODE:

class ReLUSquaredActivation(nn.Module):
    """
    Applies the relu^2 activation introduced in https://arxiv.org/abs/2109.08668v2
    """

    def forward(self, input):
        relu_applied = nn.functional.relu(input)
        squared = torch.square(relu_applied)


SUMMARY:
**Summary**

`ReLUSquaredActivation` class applies the **ReLU^2** activation function to input data. The primary purpose is to map negative values to zero, similar to ReLU.

**Key Implementation Details:**

* Applies the `ReLU` (Rectified Linear Unit) activation function via `nn.functional.relu(input)`
* Computes the square of the result using `torch.square(relu_applied)`

**Notable Dependencies/Requirements:**

* `nn.Module`
* `nn.functional.ReLU`
* `torch.square`


key: chroma:document
string_value:
CODE:

class ClassInstantier(OrderedDict):
    def __getitem__(self, key):
        content = super().__getitem__(key)
        cls, kwargs = content if isinstance(content, tuple) else (content, {})


SUMMARY:
**Summary:**

ClassInstantier is an implementation of OrderedDict that instantiates class names with keyword arguments when retrieved.

**Key Implementation Details:**

* The `__getitem__` method overrides the base dictionary behavior, utilizing the `super().__getitem__(key)` call to retrieve data.
* It checks if the content is a tuple (i.e., `content = super().__getitem__(key)`) and separates it into class name (`cls`) and keyword arguments (`kwargs`).
* If the content is not a tuple, it assigns the content as the class name and an empty dictionary as keyword arguments.

**Notable Dependencies and Requirements:**

* References:
	+ `super()`
* External Dependencies:
	+ Python's built-in `OrderedDict`
* Inherited or Implemented Interfaces:
	+ None


key: chroma:document
string_value:
CODE:

def get_activation(activation_string):
    if activation_string in ACT2FN:
        return ACT2FN[activation_string]
    else:


SUMMARY:
**Summary**

The `get_activation` function retrieves the activation function corresponding to a given string from the `ACT2FN` dictionary. If the input string is not found, it raises a `KeyError`. The primary purpose of this function is to map string representations of activation functions to their actual functions.

**Implementation Details**

- Key algorithm: Dictionary lookup
- Activation function mapping stored in `ACT2FN` dictionary

**Notable Dependencies/Requirements**

- `ACT2FN`: dictionary containing the mapping between activation function strings and their corresponding functions, e.g., `{ 'relu': npRectify }`.


key: chroma:document
string_value:
CODE:

def _gelu(x):
    """
    Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when
    initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
    0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see
    https://arxiv.org/abs/1606.08415
    """
    x = tf.convert_to_tensor(x)
    cdf = 0.5 * (1.0 + tf.math.erf(x / tf.cast(tf.sqrt(2.0), x.dtype)))



SUMMARY:
**genuine gELU Activation Function**

The `_gelu` function implements the Gaussian Error Linear Unit activation function, initially created in Google's BERT repo.

**Main Purpose and Functionality**

* Applies the gELU activation function to input `x`
* Returns the result of multiplying `x` with the cumulative distribution function (CDF) of the error function (erf)

**Key Implementation Details**

* Uses `tf.math.erf` to compute the inverse error function
* Utilizes numerical stabilization through adding 0.044715 × x³ (OpenAI GPT variant)

**Notable Dependencies and Requirements**

* `tf.convert_to_tensor(x)` converts input to TensorFlow tensor
* `math` module for mathematical functions (specifically, `sqrt` and `pi`)
* `torch` library for PyTorch functionality (`tanh`, `pow`)


key: chroma:document
string_value:
CODE:

def _gelu_new(x):
    """
    Gaussian Error Linear Unit. This is a smoother version of the GELU. Original paper: https://arxiv.org/abs/1606.0841

    Args:
        x: float Tensor to perform activation

    Returns:
        `x` with the GELU activation applied.
    """
    x = tf.convert_to_tensor(x)
    pi = tf.cast(math.pi, x.dtype)
    coeff = tf.cast(0.044715, x.dtype)
    cdf = 0.5 * (1.0 + tf.tanh(tf.sqrt(2.0 / pi) * (x + coeff * tf.pow(x, 3))))



SUMMARY:
Here's a concise RAG system summary for the provided function code chunk:

**Function Purpose:** Applies GELU activation to input tensor `x`.

**Implementation Details:**

- Uses the formula `0.5 * (1.0 + tanh(sqrt(2/pi) * (x+coeff*x^3)))` to compute the activated value.
- Utilizes `tf.tanh` for hyperbolic tangent calculation and `tf.pow` for exponentiation.

**Interfaces and Dependencies:**

* Uses tf module's `tf.convert_to_tensor`, `tf.cast`
* Importantly relies on math.pi constant from Python's built-in math library. 

Note that no external dependencies are mentioned in the given code, implying this function does not introduce any additional external modules.


key: chroma:document
string_value:
CODE:

def mish(x):
    x = tf.convert_to_tensor(x)



SUMMARY:
**Function Summary: `mish`**

The `mish` function applies the Mish activation function to the input tensor `x`. 

* Main purpose: Defines a variant of the Mish activation function, which is defined as `g(x) = x * tf.tanh(tf.math.softplus(x))`, where `tf.math.softplus(x)` is the soft-plus function applied element-wise to `x`.
* Key implementation detail: Applies the `softplus` function from TensorFlow to `x` and then multiplies the result by the hyperbolic tangent of the outcome.
* Notable dependencies:
	+ `tf` (TensorFlow)
	+ `tf.convert_to_tensor`, `tf.tanh`, `tf.math.softplus`


key: chroma:document
string_value:
CODE:

def gelu_fast(x):
    x = tf.convert_to_tensor(x)
    coeff1 = tf.cast(0.044715, x.dtype)
    coeff2 = tf.cast(0.7978845608, x.dtype)



SUMMARY:
**Function Summary: `gelu_fast`**

The `gelu_fast` function implements a simplified version of the GELU (Gaussian Error Linear Units) activation function, designed for efficient computation. The main purpose is to replace the vanilla TensorFlow implementation with a faster variant.

**Key Implementation Details:**

* Multiplies inputs by coefficients (`coeff1` and `coeff2`) for hyperbolic tangent calculation.
* Employs the `tanh` function from TensorFlow for element-wise inversion of the sigmoid function.

**Notable Dependencies/Requirements:**

* `tf` (TensorFlow): used for tensor conversions, casting, and the `tanh` activation function.


key: chroma:document
string_value:
CODE:

def quick_gelu(x):
    x = tf.convert_to_tensor(x)
    coeff = tf.cast(1.702, x.dtype)


SUMMARY:
**Summary:**

The `quick_gelu` function implements a simplified version of the GELU (Gaussian Error Linear Units) activation function. It takes a tensor input `x`, converts it to a TensorFlow tensor, scales coefficients using `tf.math.sigmoid`, and applies element-wise multiplication.

**Implementation Details:**

* Applies scaling factor `coeff` from 1.702.
* Uses `tf.math.sigmoid` to scale input coefficient.
* Multiplies scaled inputs element-wise using binary pointwise operation (`x * sigmoid_coefficient * x`).

**Notable Dependencies:**

* TensorFlow (TF) library, specifically `tf.convert_to_tensor`, `tf.cast`, and `tf.math.sigmoid`.


key: chroma:document
string_value:
CODE:

def gelu_10(x):
    """
    Clip the range of possible GeLU outputs between [-10, 10]. This is especially useful for quantization purpose, as
    it allows mapping 2 negatives values in the GeLU spectrum. For more information on this trick, please refer to
    https://arxiv.org/abs/2004.09602

    Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when
    initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
    0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see
    https://arxiv.org/abs/1606.08415 :param x: :return:
    """


SUMMARY:
**Summary**

`gelu_10(x)` is an implementation of the Gaussian Error Linear Unit (GELU) activation function. It:

* **Crops output range**: Clips GeLU outputs between [-10, 10] for quantization purposes.
* **Original implementation**: Based on an original implementation in Google's BERT repo.
* **Key details**:
	+ Uses `tf.clip_by_value` to clip the output of `_gelu(x)`.
	+ Returns a modified GeLU activation function value.

**Notable dependencies and requirements**

* TensorFlow (`tf`)
* Related class/function: `_gelu(x)`
* External dependency: TensorFlow (TF)

Note: The `_gelu(x)` function is not shown in this code snippet, but it's likely a previously implemented implementation of the GELU activation function.


key: chroma:document
string_value:
CODE:

def glu(x, axis=-1):
    """
    Gated Linear Unit. Implementation as defined in the original paper (see https://arxiv.org/abs/1612.08083), where
    the input `x` is split in two halves across a dimension (`axis`), A and B, returning A * sigmoid(B).

    Args:
        `x`: float Tensor to perform activation
        `axis`: dimension across which `x` be split in half

    Returns:
        `x` with the GLU activation applied (with its size halved across the dimension `axis`).
    """
    a, b = tf.split(x, 2, axis=axis)


SUMMARY:
Here is a concise summary of the provided code chunk:

**Function Summary:**

The `glu` function applies the Gated Linear Unit (GLU) activation to a given float Tensor `x`. It splits `x` into two halves along a specified dimension (`axis`) using `tf.split`, then multiplies the first half with the sigmoid of the second half.

**Key Details:**

* Splits input tensor `x` in half across dimension `axis` using `tf.split`.
* Applies GLU activation by multiplying the first half (`a`) with the sigmoid of the second half (`b`) (`a * tf.math.sigmoid(b)`).
* Returns the modified tensor with its size halved along axis `axis`.

**Notable Dependencies/Requirements:**

* TensorFlow library (`tf` namespace)
* Sigmoid function from `tf.math` module
* Integer division not explicitly demonstrated, but assumed as part of tensor splitting functionality.


key: chroma:document
string_value:
CODE:

def get_tf_activation(activation_string):
    if activation_string in ACT2FN:
        return ACT2FN[activation_string]
    else:


SUMMARY:
**Summary:**

This function takes an `activation_string` as input and returns the corresponding activation function from the `ACT2FN` dictionary. If the string is not found in the dictionary, it raises a KeyError with information about the missing function.

**Key Implementation Details:**

* Uses a dictionary lookup (`ACT2FN[activation_string]`) to retrieve the activation function.
* Raises a KeyError if the activation string is not found in the dictionary.

**Notable Dependencies and Requirements:**

* None

Note: The code assumes that `ACT2FN` is an existing dictionary predefined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def _get_unpad_data(attention_mask: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, int]:
    """
    Retrieves indexing data required to repad unpadded (ragged) tensors.

    Arguments:
        attention_mask (`torch.Tensor`):
            Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid.

    Return:
        indices (`torch.Tensor`):
            The indices of non-masked tokens from the flattened input sequence.
        cu_seqlens (`torch.Tensor`):
            The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,).
        max_seqlen_in_batch (`int`):
            Maximum sequence length in batch.
    """
    seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
    indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
    max_seqlen_in_batch = seqlens_in_batch.max().item()
    cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.int32), (1, 0))
    return (
        indices,
        cu_seqlens,
        max_seqlen_in_batch,


SUMMARY:
**Summary:**

 `_get_unpad_data` calculates indexing data for repadding unpadded tensors.

This function takes an `attention_mask` tensor as input and returns three outputs:

1. **indices**: non-masked token indices from the flattened input sequence (Shape: `(num_samples,)`)
2. **cu_seqlens**: cumulative sequence lengths with padded boundaries (Shape: `(batch_size + 1,)`)
3. **max_seqlen_in_batch**: maximum sequence length in batch (Type: `int`)

**Implementation Details:**

* Calculates sequence lengths using `attention_mask.sum(dim=-1)`
* Extracts non-masked token indices using `torch.nonzero`
* Computes cumulative sequence lengths with padded boundaries using `F.pad` and `torch.cumsum`
* Returns the calculated outputs

**Notable Dependencies/Requirements:**

* `torch.Tensor`
* `torch.noise`
* `torch.nn.functional`, specifically `F.pad`


key: chroma:document
string_value:
CODE:

def _upad_input(
    query_layer: torch.Tensor,
    key_layer: torch.Tensor,
    value_layer: torch.Tensor,
    attention_mask: torch.Tensor,
    query_length: int,
):
    """
    Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches.

    This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary
    tensors for query, key, value tensors.

    Arguments:
        query_layer (`torch.Tensor`):
            Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim).
        key_layer (`torch.Tensor`):
            Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
        value_layer (`torch.Tensor`):
            Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
        attention_mask (`torch.Tensor`):
            Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid.
        query_length (`int`):
            Target length.

    Return:
        query_layer (`torch.Tensor`):
            Query state without padding. Shape: (total_target_length, num_heads, head_dim).
        key_layer (`torch.Tensor`):
            Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
        value_layer (`torch.Tensor`):
            Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
        indices_q (`torch.Tensor`):
            The indices of non-masked tokens from the flattened input target sequence.
        (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`):
            The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,).
        (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`):
            Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value).
    """
    indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(attention_mask)
    batch_size, kv_seq_len, num_key_value_heads, head_dim = key_layer.shape

    key_layer = index_first_axis(key_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k)
    value_layer = index_first_axis(
        value_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
    )
    if query_length == kv_seq_len:
        query_layer = index_first_axis(query_layer.reshape(batch_size * kv_seq_len, -1, head_dim), indices_k)
        cu_seqlens_q = cu_seqlens_k
        max_seqlen_in_batch_q = max_seqlen_in_batch_k
        indices_q = indices_k
    elif query_length == 1:
        max_seqlen_in_batch_q = 1
        cu_seqlens_q = torch.arange(
            batch_size + 1, dtype=torch.int32, device=query_layer.device
        )  # There is a memcpy here, that is very bad.
        indices_q = cu_seqlens_q[:-1]
        query_layer = query_layer.squeeze(1)
    else:
        # The -q_len: slice assumes left padding.
        attention_mask = attention_mask[:, -query_length:]
        query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(query_layer, attention_mask)

    return (
        query_layer,
        key_layer,
        value_layer,
        indices_q,
        (cu_seqlens_q, cu_seqlens_k),
        (max_seqlen_in_batch_q, max_seqlen_in_batch_k),


SUMMARY:
**Summary**

This function, `_upad_input`, is used to unpadd query, key, and value tensors for a single attention operation. It takes in four input tensors: `query_layer`, `key_layer`, `value_layer`, and `attention_mask`. The function returns the unpadded tensors along with additional metadata such as sequence lengths and indices.

**Key Features**

* Unpads query, key, and value tensors using a single dimension for all tokens.
* Avoids recomputing intermediary tensors for query, key, and value using flash_attn.bert_padding.unpad_input.
* Handles case where query length is equal to the number of key/value sequences (KV) or 1.

**Implementation Details**

* Uses `_get_unpad_data` function to extract sequence lengths and indices from `attention_mask`.
* Reshapes tensors to flatten them along the batch and KV axis.
* Applies indexing using `indices_k` to move padding tokens out of the way.
* Handles cases where query length is 1 by applying squeeze operation.

**Notable Dependencies/Requirements**

* `torch.Tensor`
* _get_unpad_data function (not shown in this snippet)
* flash_attn.bert_padding.unpad_input (used for comparison)


key: chroma:document
string_value:
CODE:

def prepare_fa2_from_position_ids(query, key, value, position_ids):
    """
    This function returns necessary arguments to call `flash_attn_varlen_func`.
    All three query, key, value states will be flattened.
    Cummulative lengths of each examples in the batch will be extracted from position_ids.

    NOTE: ideally cummulative lengths should be prepared at the data collator stage

    Arguments:
        query (`torch.Tensor`):
            Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim).
        key (`torch.Tensor`):
            Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
        value (`torch.Tensor`):
            Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim).
        position_ids (`torch.Tensor`):
            Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid.

    Return:
        query (`torch.Tensor`):
            Query state without padding. Shape: (total_target_length, num_heads, head_dim).
        key (`torch.Tensor`):
            Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
        value (`torch.Tensor`):
            Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim).
        indices_q (`torch.Tensor`):
            The indices of non-masked tokens from the flattened input target sequence.
        (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`):
            The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,).
        (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`):
            Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value).
    """
    query = query.view(-1, query.size(-2), query.size(-1))
    key = key.contiguous().view(-1, key.size(-2), key.size(-1))
    value = value.contiguous().view(-1, value.size(-2), value.size(-1))
    position_ids = position_ids.flatten()
    indices_q = torch.arange(position_ids.size(0), device=position_ids.device, dtype=torch.int32)

    cu_seq_lens = torch.cat(
        (
            indices_q[position_ids == 0],
            torch.tensor(position_ids.size(), device=position_ids.device, dtype=torch.int32),
        )
    )

    max_length = position_ids.max() + 1



SUMMARY:
**Summary:**

This function `prepare_fa2_from_position_ids` prepares necessary arguments for the `flash_attn_varlen_func` by flattening input tensors and extracting cumulative sequence lengths from `position_ids`.

**Key Details:**

* Flattens query, key, value tensors with padding.
* Extracts indices of non-masked tokens from query using `torch.arange`.
* Computes cumulative sequence lengths (cu_seq_lens) for both target (query) and source (key, value).
* Calculates maximum sequence length in batch (`max_seqlen_in_batch_q` for query, `max_seqlen_in_batch_k` for key/value).

**Return Values:**

* Unpadded query tensor (`torch.Tensor`)
* Unpadded key tensor (`torch.Tensor`)
* Unpadded value tensor (`torch.Tensor`)
* Indices of non-masked tokens in query tensor (`torch.Tensor`)
* Cumulative sequence lengths tuple (`(cu_seq_lens, cu_seq_lens)`)
* Maximum sequence length tuples (`(max_seqlen_in_batch_q, max_seqlen_in_batch_k)`)


key: chroma:document
string_value:
CODE:

def fa_peft_integration_check(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    target_dtype: Optional[torch.dtype] = None,
):
    """
    PEFT usually casts the layer norms in float32 for training stability reasons
    therefore the input hidden states gets silently casted in float32. Hence, we need
    cast them back in float16 / bfloat16 just to be sure everything works as expected.
    This might slowdown training & inference so it is recommended to not cast the LayerNorms!

    Args:
        query (`torch.Tensor`):
            Input query states to be passed to Flash Attention API
        key (`torch.Tensor`):
            Input key states to be passed to Flash Attention API
        value (`torch.Tensor`):
            Input value states to be passed to Flash Attention API
        target_dtype (`torch.dtype`, *optional*):
            The dtype to convert the attention tensors to. Conversion can be ignored by
            not providing the target dtype.
    """
    if target_dtype is None:
        return query, key, value

    input_dtype = value.dtype
    if input_dtype == torch.float32:
        logger.warning_once(
            f"The input hidden states seems to be silently casted in float32, this might be related to"
            f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in"
            f" {target_dtype}."
        )

        query = query.to(target_dtype)
        key = key.to(target_dtype)
        value = value.to(target_dtype)



SUMMARY:
**Function Summary: `fa_peft_integration_check`**

**Purpose:** Casts input tensors (`query`, `key`, `value`) to a specified target data type (`target_dtype`) in the PEFT integration check.

**Key Implementation Details:**

* Casts input tensors to `float16/bfloat16` if input is already in `torch.float32` for stability reasons.
* Supports optional `target_dtype` conversion, which can be ignored for compatibility.
* Uses `logger.warning_once` to log potential issues related to silent casting.

**Method Signatures and Interfaces:**
```python
import torch as torch

def fa_peft_integration_check(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    target_dtype: Optional[torch.dtype] = None,
):
```

**External Dependencies:**

* `torch` for tensor operations, including data type conversions.
* `logger` for logging warnings (not shown in this code snippet).


key: chroma:document
string_value:
CODE:

def _flash_attention_forward(
    query_states: torch.Tensor,
    key_states: torch.Tensor,
    value_states: torch.Tensor,
    attention_mask: torch.Tensor,
    query_length: int,
    is_causal: bool,
    dropout: float = 0.0,
    position_ids: Optional[torch.Tensor] = None,
    softmax_scale: Optional[float] = None,
    sliding_window: Optional[int] = None,
    use_top_left_mask: bool = False,
    softcap: Optional[float] = None,
    deterministic: bool = None,
    cu_seq_lens_q: Optional[torch.LongTensor] = None,
    cu_seq_lens_k: Optional[torch.LongTensor] = None,
    max_length_q: Optional[int] = None,
    max_length_k: Optional[int] = None,
    target_dtype: Optional[torch.dtype] = None,
    **kwargs,
):
    """
    Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token
    first unpad the input, then computes the attention scores and pad the final attention scores.

    Args:
        query_states (`torch.Tensor`):
            Input query states to be passed to Flash Attention API
        key_states (`torch.Tensor`):
            Input key states to be passed to Flash Attention API
        value_states (`torch.Tensor`):
            Input value states to be passed to Flash Attention API
        attention_mask (`torch.Tensor`):
            The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the
            position of padding tokens and 1 for the position of non-padding tokens.
        dropout (`float`):
            Attention dropout
        softmax_scale (`float`, *optional*):
            The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim)
        use_top_left_mask (`bool`, defaults to `False`):
            flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference.
        softcap (`float`, *optional*):
            Softcap for the attention logits, used e.g. in gemma2.
        deterministic (`bool`, *optional*):
            Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled.
    """
    if not use_top_left_mask:
        causal = is_causal
    else:
        # TODO: Remove the `query_length != 1` check once Flash Attention for RoCm is bumped to 2.1.
        causal = is_causal and query_length != 1

    # Assuming 4D tensors, key_states.shape[1] is the key/value sequence length (source length).
    use_sliding_windows = (
        _flash_supports_window_size and sliding_window is not None and key_states.shape[1] > sliding_window
    )
    flash_kwargs = {"window_size": (sliding_window, sliding_window)} if use_sliding_windows else {}

    if flash_241:
        if deterministic is None:
            deterministic = deterministic_g
        flash_kwargs["deterministic"] = deterministic

    if softcap is not None:
        flash_kwargs["softcap"] = softcap

    # PEFT possibly silently casts tensors to fp32, this potentially reconverts to correct dtype or is a no op
    query_states, key_states, value_states = fa_peft_integration_check(
        query_states, key_states, value_states, target_dtype
    )

    # Contains at least one padding token in the sequence
    if attention_mask is not None:
        batch_size = query_states.shape[0]
        query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = _upad_input(
            query_states, key_states, value_states, attention_mask, query_length
        )
        cu_seqlens_q, cu_seqlens_k = cu_seq_lens
        max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens

        attn_output_unpad = flash_attn_varlen_func(
            query_states,
            key_states,
            value_states,
            cu_seqlens_q=cu_seqlens_q,
            cu_seqlens_k=cu_seqlens_k,
            max_seqlen_q=max_seqlen_in_batch_q,
            max_seqlen_k=max_seqlen_in_batch_k,
            dropout_p=dropout,
            softmax_scale=softmax_scale,
            causal=causal,
            **flash_kwargs,
        )
        attn_output = pad_input(attn_output_unpad, indices_q, batch_size, query_length)

    # If position_ids is provided and check all examples do not contain only 1 sequence, If tensor in increasing
    # then we probably have one sequence, otherwise it is packed. Additionally check we are in pre-fill/training stage.
    # Use `flash_attn_varlen_func` to prevent cross-example attention and also allow padding free approach
    elif position_ids is not None and (
        max_length_q is not None or (query_length != 1 and not (torch.diff(position_ids, dim=-1) >= 0).all())
    ):
        batch_size = query_states.size(0)

        if cu_seq_lens_q is None or cu_seq_lens_k is None:
            query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = (
                prepare_fa2_from_position_ids(query_states, key_states, value_states, position_ids)
            )

            cu_seq_lens_q, cu_seq_lens_k = cu_seq_lens
            max_length_q, max_length_k = max_seq_lens

        else:
            query_states = query_states.reshape(-1, query_states.size(-2), query_states.size(-1))
            key_states = key_states.reshape(-1, key_states.size(-2), key_states.size(-1))
            value_states = value_states.reshape(-1, value_states.size(-2), value_states.size(-1))

        attn_output = flash_attn_varlen_func(
            query_states,
            key_states,
            value_states,
            cu_seqlens_q=cu_seq_lens_q,
            cu_seqlens_k=cu_seq_lens_k,
            max_seqlen_q=max_length_q,
            max_seqlen_k=max_length_k,
            dropout_p=dropout,
            softmax_scale=softmax_scale,
            causal=causal,
            **flash_kwargs,
        )

        attn_output = attn_output.view(batch_size, -1, attn_output.size(-2), attn_output.size(-1))

    else:
        attn_output = flash_attn_func(
            query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=causal, **flash_kwargs
        )



SUMMARY:
**Function Summary: `_flash_attention_forward`**

**Purpose and Functionality:**
The `_flash_attention_forward` function implements the forward pass of the Flash Attention mechanism in transformer models. It takes query, key, value states, an attention mask, and optional parameters such as dropout rate, softmax scaling factor, and position IDs.

**Key Implementation Details:**

* The function checks for the presence of padding tokens in the input sequences and un pads them if necessary.
* It uses the `flash_attn` function to compute attention scores and the `pad_input` function to pad the final attention output.
* If position IDs are provided, it uses a variant of the Flash Attention mechanism that allows for packing multiple sequences into a single tensor.

**Notable Dependencies and Requirements:**

* `torch` library for tensor operations
* `flash_attn` function, which is not shown in this code snippet
* `_upad_input`, `fa_peft_integration_check`, and `prepare_fa2_from_position_ids` functions, which are not shown in this code snippet

**Method Signatures and Interfaces:**

* `flash_attn_varlen_func`: computes attention scores with variable-length sequences.
* `pad_input`: pads the final attention output to ensure it has a specific shape.

Note that the exact method signatures and interfaces used by these functions are not explicitly stated in the provided code snippet.


key: chroma:document
string_value:
CODE:

class FlashAttentionKwargs(TypedDict, total=False):
    """
    Keyword arguments for Flash Attention with Compile.

    Attributes:
        cu_seq_lens_q (`torch.LongTensor`, *optional*)
            Gets cumlative sequence length for query state.
        cu_seq_lens_k (`torch.LongTensor`, *optional*)
            Gets cumlative sequence length for key state.
        max_length_q (`int`, *optional*):
            Maximum sequence length for query state.
        max_length_k (`int`, *optional*):
            Maximum sequence length for key state.
    """

    cu_seq_lens_q: Optional[torch.LongTensor]
    cu_seq_lens_k: Optional[torch.LongTensor]
    max_length_q: Optional[int]


SUMMARY:
**Summary**

The `FlashAttentionKwargs` class defines keyword arguments for Flash Attention, a neural network architecture compound (Cumlab). It contains optional attributes:

*   Cumulative sequence lengths for query and key states (`cu_seq_lens_q` and `cu_seq_lens_k`)
*   Maximum allowed sequence lengths for query and key states (`max_length_q` and `max_length_k`)

**Key Implementation Details**

*   Uses `torch(LongTensor)` as data type for cumulative sequence lengths
*   Enforces optional attribute usage (i.e., variables are labeled as *optional*)

**Notable Dependencies or Requirements**

*   Requires the PyTorch library (`torch`)
*   Inherently uses Cumlab's Compile feature (not explicitly mentioned but implied by class name and purpose)


key: docstring
string_value:
# TODO: Move BatchFeature to be imported by both image_processing_utils and image_processing_utils
# We override the class string here, but logic is the same.


key: chroma:document
string_value:
CODE:
# We override the class string here, but logic is the same.
class BatchFeature(BaseBatchFeature):
    r"""
    Holds the output of the image processor specific `__call__` methods.

    This class is derived from a python dictionary and can be used as a dictionary.

    Args:
        data (`dict`):
            Dictionary of lists/arrays/tensors returned by the __call__ method ('pixel_values', etc.).
        tensor_type (`Union[None, str, TensorType]`, *optional*):
            You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at
            initialization.


SUMMARY:
**Summary:**

`BatchFeature` class:

- Derived from `BaseBatchFeature` (class inherited).
- Represents output of image processor's `__call__` methods as a dictionary.
- Accepts `data` (dictionary) and optional `tensor_type` for tensor conversion during initialization.

**Key Implementation Detail:**
The key implementation detail is the override of the class string, but actual logic remains in the parent class (`BaseBatchFeature`).

**Notable Dependency/Requirement:**
- `BaseBatchFeature`: Inherited from.
- `Union[None, str, TensorType]`: Optional tensor type requirement.

**External Dependencies:**

- `image_processing_utils`: Import module for the class.


key: chroma:document
string_value:
CODE:
# TODO: (Amy) - factor out the common parts of this and the feature extractor
class ImageProcessingMixin(PushToHubMixin):
    """
    This is an image processor mixin used to provide saving/loading functionality for sequential and image feature
    extractors.
    """

    _auto_class = None

    def __init__(self, **kwargs):
        """Set elements of `kwargs` as attributes."""
        # This key was saved while we still used `XXXFeatureExtractor` for image processing. Now we use
        # `XXXImageProcessor`, this attribute and its value are misleading.
        kwargs.pop("feature_extractor_type", None)
        # Pop "processor_class" as it should be saved as private attribute
        self._processor_class = kwargs.pop("processor_class", None)
        # Additional attributes without default values
        for key, value in kwargs.items():
            try:
                setattr(self, key, value)
            except AttributeError as err:
                logger.error(f"Can't set {key} with value {value} for {self}")
                raise err

    def _set_processor_class(self, processor_class: str):
        """Sets processor class as an attribute."""
        self._processor_class = processor_class

    @classmethod
    def from_pretrained(
        cls: Type[ImageProcessorType],
        pretrained_model_name_or_path: Union[str, os.PathLike],
        cache_dir: Optional[Union[str, os.PathLike]] = None,
        force_download: bool = False,
        local_files_only: bool = False,
        token: Optional[Union[str, bool]] = None,
        revision: str = "main",
        **kwargs,
    ) -> ImageProcessorType:
        r"""
        Instantiate a type of [`~image_processing_utils.ImageProcessingMixin`] from an image processor.

        Args:
            pretrained_model_name_or_path (`str` or `os.PathLike`):
                This can be either:

                - a string, the *model id* of a pretrained image_processor hosted inside a model repo on
                  huggingface.co.
                - a path to a *directory* containing a image processor file saved using the
                  [`~image_processing_utils.ImageProcessingMixin.save_pretrained`] method, e.g.,
                  `./my_model_directory/`.
                - a path or url to a saved image processor JSON *file*, e.g.,
                  `./my_model_directory/preprocessor_config.json`.
            cache_dir (`str` or `os.PathLike`, *optional*):
                Path to a directory in which a downloaded pretrained model image processor should be cached if the
                standard cache should not be used.
            force_download (`bool`, *optional*, defaults to `False`):
                Whether or not to force to (re-)download the image processor files and override the cached versions if
                they exist.
            resume_download:
                Deprecated and ignored. All downloads are now resumed by default when possible.
                Will be removed in v5 of Transformers.
            proxies (`Dict[str, str]`, *optional*):
                A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
                'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
            token (`str` or `bool`, *optional*):
                The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use
                the token generated when running `huggingface-cli login` (stored in `~/.huggingface`).
            revision (`str`, *optional*, defaults to `"main"`):
                The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
                git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
                identifier allowed by git.


                <Tip>

                To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>"`.

                </Tip>

            return_unused_kwargs (`bool`, *optional*, defaults to `False`):
                If `False`, then this function returns just the final image processor object. If `True`, then this
                functions returns a `Tuple(image_processor, unused_kwargs)` where *unused_kwargs* is a dictionary
                consisting of the key/value pairs whose keys are not image processor attributes: i.e., the part of
                `kwargs` which has not been used to update `image_processor` and is otherwise ignored.
            subfolder (`str`, *optional*, defaults to `""`):
                In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
                specify the folder name here.
            kwargs (`Dict[str, Any]`, *optional*):
                The values in kwargs of any keys which are image processor attributes will be used to override the
                loaded values. Behavior concerning key/value pairs whose keys are *not* image processor attributes is
                controlled by the `return_unused_kwargs` keyword parameter.

        Returns:
            A image processor of type [`~image_processing_utils.ImageProcessingMixin`].

        Examples:

        ```python
        # We can't instantiate directly the base class *ImageProcessingMixin* so let's show the examples on a
        # derived class: *CLIPImageProcessor*
        image_processor = CLIPImageProcessor.from_pretrained(
            "openai/clip-vit-base-patch32"
        )  # Download image_processing_config from huggingface.co and cache.
        image_processor = CLIPImageProcessor.from_pretrained(
            "./test/saved_model/"
        )  # E.g. image processor (or model) was saved using *save_pretrained('./test/saved_model/')*
        image_processor = CLIPImageProcessor.from_pretrained("./test/saved_model/preprocessor_config.json")
        image_processor = CLIPImageProcessor.from_pretrained(
            "openai/clip-vit-base-patch32", do_normalize=False, foo=False
        )
        assert image_processor.do_normalize is False
        image_processor, unused_kwargs = CLIPImageProcessor.from_pretrained(
            "openai/clip-vit-base-patch32", do_normalize=False, foo=False, return_unused_kwargs=True
        )
        assert image_processor.do_normalize is False
        assert unused_kwargs == {"foo": False}
        ```"""
        kwargs["cache_dir"] = cache_dir
        kwargs["force_download"] = force_download
        kwargs["local_files_only"] = local_files_only
        kwargs["revision"] = revision

        use_auth_token = kwargs.pop("use_auth_token", None)
        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token

        if token is not None:
            kwargs["token"] = token

        image_processor_dict, kwargs = cls.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)

        return cls.from_dict(image_processor_dict, **kwargs)

    def save_pretrained(self, save_directory: Union[str, os.PathLike], push_to_hub: bool = False, **kwargs):
        """
        Save an image processor object to the directory `save_directory`, so that it can be re-loaded using the
        [`~image_processing_utils.ImageProcessingMixin.from_pretrained`] class method.

        Args:
            save_directory (`str` or `os.PathLike`):
                Directory where the image processor JSON file will be saved (will be created if it does not exist).
            push_to_hub (`bool`, *optional*, defaults to `False`):
                Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
                repository you want to push to with `repo_id` (will default to the name of `save_directory` in your
                namespace).
            kwargs (`Dict[str, Any]`, *optional*):
                Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.
        """
        use_auth_token = kwargs.pop("use_auth_token", None)

        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if kwargs.get("token", None) is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            kwargs["token"] = use_auth_token

        if os.path.isfile(save_directory):
            raise AssertionError(f"Provided path ({save_directory}) should be a directory, not a file")

        os.makedirs(save_directory, exist_ok=True)

        if push_to_hub:
            commit_message = kwargs.pop("commit_message", None)
            repo_id = kwargs.pop("repo_id", save_directory.split(os.path.sep)[-1])
            repo_id = self._create_repo(repo_id, **kwargs)
            files_timestamps = self._get_files_timestamps(save_directory)

        # If we have a custom config, we copy the file defining it in the folder and set the attributes so it can be
        # loaded from the Hub.
        if self._auto_class is not None:
            custom_object_save(self, save_directory, config=self)

        # If we save using the predefined names, we can load using `from_pretrained`
        output_image_processor_file = os.path.join(save_directory, IMAGE_PROCESSOR_NAME)

        self.to_json_file(output_image_processor_file)
        logger.info(f"Image processor saved in {output_image_processor_file}")

        if push_to_hub:
            self._upload_modified_files(
                save_directory,
                repo_id,
                files_timestamps,
                commit_message=commit_message,
                token=kwargs.get("token"),
            )

        return [output_image_processor_file]

    @classmethod
    def get_image_processor_dict(
        cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs
    ) -> Tuple[Dict[str, Any], Dict[str, Any]]:
        """
        From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a
        image processor of type [`~image_processor_utils.ImageProcessingMixin`] using `from_dict`.

        Parameters:
            pretrained_model_name_or_path (`str` or `os.PathLike`):
                The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
            subfolder (`str`, *optional*, defaults to `""`):
                In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
                specify the folder name here.
            image_processor_filename (`str`, *optional*, defaults to `"config.json"`):
                The name of the file in the model directory to use for the image processor config.

        Returns:
            `Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the image processor object.
        """
        cache_dir = kwargs.pop("cache_dir", None)
        force_download = kwargs.pop("force_download", False)
        resume_download = kwargs.pop("resume_download", None)
        proxies = kwargs.pop("proxies", None)
        token = kwargs.pop("token", None)
        use_auth_token = kwargs.pop("use_auth_token", None)
        local_files_only = kwargs.pop("local_files_only", False)
        revision = kwargs.pop("revision", None)
        subfolder = kwargs.pop("subfolder", "")
        image_processor_filename = kwargs.pop("image_processor_filename", IMAGE_PROCESSOR_NAME)

        from_pipeline = kwargs.pop("_from_pipeline", None)
        from_auto_class = kwargs.pop("_from_auto", False)

        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token

        user_agent = {"file_type": "image processor", "from_auto_class": from_auto_class}
        if from_pipeline is not None:
            user_agent["using_pipeline"] = from_pipeline

        if is_offline_mode() and not local_files_only:
            logger.info("Offline mode: forcing local_files_only=True")
            local_files_only = True

        pretrained_model_name_or_path = str(pretrained_model_name_or_path)
        is_local = os.path.isdir(pretrained_model_name_or_path)
        if os.path.isdir(pretrained_model_name_or_path):
            image_processor_file = os.path.join(pretrained_model_name_or_path, image_processor_filename)
        if os.path.isfile(pretrained_model_name_or_path):
            resolved_image_processor_file = pretrained_model_name_or_path
            is_local = True
        elif is_remote_url(pretrained_model_name_or_path):
            image_processor_file = pretrained_model_name_or_path
            resolved_image_processor_file = download_url(pretrained_model_name_or_path)
        else:
            image_processor_file = image_processor_filename
            try:
                # Load from local folder or from cache or download from model Hub and cache
                resolved_image_processor_file = cached_file(
                    pretrained_model_name_or_path,
                    image_processor_file,
                    cache_dir=cache_dir,
                    force_download=force_download,
                    proxies=proxies,
                    resume_download=resume_download,
                    local_files_only=local_files_only,
                    token=token,
                    user_agent=user_agent,
                    revision=revision,
                    subfolder=subfolder,
                )
            except EnvironmentError:
                # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted to
                # the original exception.
                raise
            except Exception:
                # For any other exception, we throw a generic error.
                raise EnvironmentError(
                    f"Can't load image processor for '{pretrained_model_name_or_path}'. If you were trying to load"
                    " it from 'https://huggingface.co/models', make sure you don't have a local directory with the"
                    f" same name. Otherwise, make sure '{pretrained_model_name_or_path}' is the correct path to a"
                    f" directory containing a {image_processor_filename} file"
                )

        try:
            # Load image_processor dict
            with open(resolved_image_processor_file, "r", encoding="utf-8") as reader:
                text = reader.read()
            image_processor_dict = json.loads(text)

        except json.JSONDecodeError:
            raise EnvironmentError(
                f"It looks like the config file at '{resolved_image_processor_file}' is not a valid JSON file."
            )

        if is_local:
            logger.info(f"loading configuration file {resolved_image_processor_file}")
        else:
            logger.info(
                f"loading configuration file {image_processor_file} from cache at {resolved_image_processor_file}"
            )
            if "auto_map" in image_processor_dict:
                image_processor_dict["auto_map"] = add_model_info_to_auto_map(
                    image_processor_dict["auto_map"], pretrained_model_name_or_path
                )
            if "custom_pipelines" in image_processor_dict:
                image_processor_dict["custom_pipelines"] = add_model_info_to_custom_pipelines(
                    image_processor_dict["custom_pipelines"], pretrained_model_name_or_path
                )

        return image_processor_dict, kwargs

    @classmethod
    def from_dict(cls, image_processor_dict: Dict[str, Any], **kwargs):
        """
        Instantiates a type of [`~image_processing_utils.ImageProcessingMixin`] from a Python dictionary of parameters.

        Args:
            image_processor_dict (`Dict[str, Any]`):
                Dictionary that will be used to instantiate the image processor object. Such a dictionary can be
                retrieved from a pretrained checkpoint by leveraging the
                [`~image_processing_utils.ImageProcessingMixin.to_dict`] method.
            kwargs (`Dict[str, Any]`):
                Additional parameters from which to initialize the image processor object.

        Returns:
            [`~image_processing_utils.ImageProcessingMixin`]: The image processor object instantiated from those
            parameters.
        """
        image_processor_dict = image_processor_dict.copy()
        return_unused_kwargs = kwargs.pop("return_unused_kwargs", False)

        # The `size` parameter is a dict and was previously an int or tuple in feature extractors.
        # We set `size` here directly to the `image_processor_dict` so that it is converted to the appropriate
        # dict within the image processor and isn't overwritten if `size` is passed in as a kwarg.
        if "size" in kwargs and "size" in image_processor_dict:
            image_processor_dict["size"] = kwargs.pop("size")
        if "crop_size" in kwargs and "crop_size" in image_processor_dict:
            image_processor_dict["crop_size"] = kwargs.pop("crop_size")

        image_processor = cls(**image_processor_dict)

        # Update image_processor with kwargs if needed
        to_remove = []
        for key, value in kwargs.items():
            if hasattr(image_processor, key):
                setattr(image_processor, key, value)
                to_remove.append(key)
        for key in to_remove:
            kwargs.pop(key, None)

        logger.info(f"Image processor {image_processor}")
        if return_unused_kwargs:
            return image_processor, kwargs
        else:
            return image_processor

    def to_dict(self) -> Dict[str, Any]:
        """
        Serializes this instance to a Python dictionary.

        Returns:
            `Dict[str, Any]`: Dictionary of all the attributes that make up this image processor instance.
        """
        output = copy.deepcopy(self.__dict__)
        output["image_processor_type"] = self.__class__.__name__

        return output

    @classmethod
    def from_json_file(cls, json_file: Union[str, os.PathLike]):
        """
        Instantiates a image processor of type [`~image_processing_utils.ImageProcessingMixin`] from the path to a JSON
        file of parameters.

        Args:
            json_file (`str` or `os.PathLike`):
                Path to the JSON file containing the parameters.

        Returns:
            A image processor of type [`~image_processing_utils.ImageProcessingMixin`]: The image_processor object
            instantiated from that JSON file.
        """
        with open(json_file, "r", encoding="utf-8") as reader:
            text = reader.read()
        image_processor_dict = json.loads(text)
        return cls(**image_processor_dict)

    def to_json_string(self) -> str:
        """
        Serializes this instance to a JSON string.

        Returns:
            `str`: String containing all the attributes that make up this feature_extractor instance in JSON format.
        """
        dictionary = self.to_dict()

        for key, value in dictionary.items():
            if isinstance(value, np.ndarray):
                dictionary[key] = value.tolist()

        # make sure private name "_processor_class" is correctly
        # saved as "processor_class"
        _processor_class = dictionary.pop("_processor_class", None)
        if _processor_class is not None:
            dictionary["processor_class"] = _processor_class

        return json.dumps(dictionary, indent=2, sort_keys=True) + "\n"

    def to_json_file(self, json_file_path: Union[str, os.PathLike]):
        """
        Save this instance to a JSON file.

        Args:
            json_file_path (`str` or `os.PathLike`):
                Path to the JSON file in which this image_processor instance's parameters will be saved.
        """
        with open(json_file_path, "w", encoding="utf-8") as writer:
            writer.write(self.to_json_string())

    def __repr__(self):
        return f"{self.__class__.__name__} {self.to_json_string()}"

    @classmethod
    def register_for_auto_class(cls, auto_class="AutoImageProcessor"):
        """
        Register this class with a given auto class. This should only be used for custom image processors as the ones
        in the library are already mapped with `AutoImageProcessor `.

        <Tip warning={true}>

        This API is experimental and may have some slight breaking changes in the next releases.

        </Tip>

        Args:
            auto_class (`str` or `type`, *optional*, defaults to `"AutoImageProcessor "`):
                The auto class to register this new image processor with.
        """
        if not isinstance(auto_class, str):
            auto_class = auto_class.__name__

        import transformers.models.auto as auto_module

        if not hasattr(auto_module, auto_class):
            raise ValueError(f"{auto_class} is not a valid auto class.")

        cls._auto_class = auto_class

    def fetch_images(self, image_url_or_urls: Union[str, List[str]]):
        """
        Convert a single or a list of urls into the corresponding `PIL.Image` objects.

        If a single url is passed, the return value will be a single object. If a list is passed a list of objects is
        returned.
        """
        headers = {
            "User-Agent": (
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0"
                " Safari/537.36"
            )
        }
        if isinstance(image_url_or_urls, list):
            return [self.fetch_images(x) for x in image_url_or_urls]
        elif isinstance(image_url_or_urls, str):
            response = requests.get(image_url_or_urls, stream=True, headers=headers)
            response.raise_for_status()
            return Image.open(BytesIO(response.content))
        else:


SUMMARY:
The code provided is for an image processor class in Python, specifically designed to work with the TensorFlow library. It provides methods for processing images, such as resizing, rotating, and flipping, as well as registering itself with a given auto-class.

Here are some key features of this class:

1.  **Initialization**: The class can be initialized with various parameters like image dimensions, cropping, resampling filters etc.
2.  **Image Preprocessing**: It offers methods for preprocessing images such as resizing, rotating, and flipping.
3.  **Registration with Auto-Classes**: This class supports registration with a given auto-class. This is useful when working with custom image processors.

Here is an example of how you might use this class to resize an image by 40%:
```python
from tensorflow.keras.preprocessing.image import load_img, img_to_array

img = load_img("path_to_your_image.jpg", target_size=None)
img = img.resize((int(img.size[0] * 0.4), int(img.size[1] * 0.4)))
img_array = load_img(img, target_size=(224, 224))
print(img_array.shape)
```
This code resizes the image by a factor of 40% (i.e., it crops from the top-right or bottom-left corner 30%) and then converts it to the (224, 224) shape.

**Update:** The following example demonstrates registering this class with an auto-class `AutoImageProcessor` :
```python
from PIL import Image
import requests

class MyImageProcessor(ImageProcessor):
    pass


class AutoImageProcessor:
    def __init__(self):
        pass

# Register the image processor
MyImageProcessor.register_for_auto_class()
```

To use this class effectively with an existing model created in the Hugging Face Transformers library, you may have to first convert the images into NumPy arrays and then load them into the specified dimensions. 

For example, if `model_name` is the name of your pre-trained model:
```python
from tensorflow.keras.applications import ResNet50
from transformers import AutoImageProcessor

# Load my model using AutoImageProcessor
my_model = models.AutoImageProcessor("resnet50")

my_mdl_1d_features = my_model.extract_features([list_to_np(image)], return_tensors="pt", size_divisor=16)
feature_extractor = my_image_processor.to_feature_importance(
    [None, None],
    max_size=3 * 512,
    feature_map_layers=(3,),
    return_as_list=True
)

# Use the model outputs to perform classification or regression...
```


key: chroma:document
string_value:
CODE:

def default_logdir() -> str:
    """
    Same default as PyTorch
    """
    import socket
    from datetime import datetime

    current_time = datetime.now().strftime("%b%d_%H-%M-%S")


SUMMARY:
**Summary:**

This function, `default_logdir()`, generates and returns a default log directory path. It combines the current date-time with a hostname-based suffix and stores it in a "runs" directory.

**Key Details:**

- **Function Signatures:** None
- **Main Purpose:** Generate default log directory path.
- **Key Implementation:** Uses `datetime.now()` for date-time, `socket.gethostname()` for hostname, and `os.path.join()` for directory construction.
- **External Dependencies:** 
  - `socket`
  - `strptime` (from `datetime`)
  - `gethostname` (from `socket`)
  - `os`
- **Interfaces/Inheritance:**
  None


key: chroma:document
string_value:
CODE:

def get_int_from_env(env_keys, default):
    """Returns the first positive env value found in the `env_keys` list or the default."""
    for e in env_keys:
        val = int(os.environ.get(e, -1))
        if val >= 0:
            return val


SUMMARY:
**Code Summary**

This function retrieves an integer value from environment variables specified by `env_keys` and returns the first positive result found. If no positive value is found, it returns the provided `default`.

**Key Implementation Details**

- Uses `os.environ.get()` to retrieve environment variable values.
- Attempts to convert each result to an integer, returning immediately on success.

**Function Signature**

`get_int_from_env(env_keys: list, default: any) -> int`

**Notable Dependencies/Requirements**

- `os`: for accessing environment variables.

No inherited or implemented interfaces noted.


key: chroma:document
string_value:
CODE:

def get_xla_device_type(device: "torch.device") -> Optional[str]:
    """
    Returns the xla device type (CPU|GPU|TPU) or None if the device is a non-xla device.
    """
    if is_torch_xla_available():
        if device.type == "cpu":
            return "CPU"
        return xm.xla_real_devices([device])[0].split(":")[0]


SUMMARY:
**Function Summary**

`get_xla_device_type(device: torch.device) -> Optional[str]`

Returns the XLA device type (CPU, GPU, TPU) or `None` for non-XLA devices.

**Key Features**

* Checks if `torch_xla` is available and returns a valid XLA device type
* Returns "CPU" if the device is a CPU
* Extracts the device type from an XLA real device (GPU/TPU) using `xm.xla_real_devices()`
* Assumes `device.type == ["cpu"]` comparison implicitly matches with string comparison

**Implementation Details**

* Uses `torch.device` object as input
* Utilizes `is_torch_xla_available()` function to check availability
* Utilizes `xm.xla_real_devices()` function to extract XLA device type

**Note**: Assumes familiarity with `torch_xla` and `xm` dependencies. No external dependencies or requirements specified.


key: chroma:document
string_value:
CODE:

class OptimizerNames(ExplicitEnum):
    """
    Stores the acceptable string identifiers for optimizers.
    """

    ADAMW_HF = "adamw_hf"
    ADAMW_TORCH = "adamw_torch"
    ADAMW_TORCH_FUSED = "adamw_torch_fused"
    ADAMW_TORCH_XLA = "adamw_torch_xla"
    ADAMW_TORCH_NPU_FUSED = "adamw_torch_npu_fused"
    ADAMW_APEX_FUSED = "adamw_apex_fused"
    ADAFACTOR = "adafactor"
    ADAMW_ANYPRECISION = "adamw_anyprecision"
    ADAMW_TORCH_4BIT = "adamw_torch_4bit"
    ADEMAMIX = "ademamix"
    SGD = "sgd"
    ADAGRAD = "adagrad"
    ADAMW_BNB = "adamw_bnb_8bit"
    ADAMW_8BIT = "adamw_8bit"  # just an alias for adamw_bnb_8bit
    ADEMAMIX_8BIT = "ademamix_8bit"
    LION_8BIT = "lion_8bit"
    LION = "lion_32bit"
    PAGED_ADAMW = "paged_adamw_32bit"
    PAGED_ADAMW_8BIT = "paged_adamw_8bit"
    PAGED_ADEMAMIX = "paged_ademamix_32bit"
    PAGED_ADEMAMIX_8BIT = "paged_ademamix_8bit"
    PAGED_LION = "paged_lion_32bit"
    PAGED_LION_8BIT = "paged_lion_8bit"
    RMSPROP = "rmsprop"
    RMSPROP_BNB = "rmsprop_bnb"
    RMSPROP_8BIT = "rmsprop_bnb_8bit"
    RMSPROP_32BIT = "rmsprop_bnb_32bit"
    GALORE_ADAMW = "galore_adamw"
    GALORE_ADAMW_8BIT = "galore_adamw_8bit"
    GALORE_ADAFACTOR = "galore_adafactor"
    GALORE_ADAMW_LAYERWISE = "galore_adamw_layerwise"
    GALORE_ADAMW_8BIT_LAYERWISE = "galore_adamw_8bit_layerwise"
    GALORE_ADAFACTOR_LAYERWISE = "galore_adafactor_layerwise"
    LOMO = "lomo"
    ADALOMO = "adalomo"
    GROKADAMW = "grokadamw"
    SCHEDULE_FREE_ADAMW = "schedule_free_adamw"


SUMMARY:
**Summary:**

`OptimizerNames` class defines an enumeration of string identifiers for various optimizer names used in machine learning models. The class is part of the `ExplicitEnum` design pattern.

**Main Purpose and Functionality:**
The class serves as a centralized registry for different optimizer names, making it easier to switch between optimizers or retrieve specific optimizer names throughout the codebase.

**Key Implementation Details:**

* The class contains 42 named constants, each representing an optimizer name (e.g., `ADAMW_HF`, `SGD`, etc.).
* The constants are divided into categories based on their type or characteristics (e.g., mixed precision optimizers like `ABN_BNB_8BIT` and integer-8-bit optimizers like `SGD`, etc.).

**Notable Dependencies or Requirements:**

* No external dependencies are mentioned.
* None of the named constants appear to inherit from a parent class or implement an interface.


key: chroma:document
string_value:
CODE:

def _convert_str_dict(passed_value: dict):
    "Safely checks that a passed value is a dictionary and converts any string values to their appropriate types."
    for key, value in passed_value.items():
        if isinstance(value, dict):
            passed_value[key] = _convert_str_dict(value)
        elif isinstance(value, str):
            # First check for bool and convert
            if value.lower() in ("true", "false"):
                passed_value[key] = value.lower() == "true"
            # Check for digit
            elif value.isdigit():
                passed_value[key] = int(value)
            elif value.replace(".", "", 1).isdigit():
                passed_value[key] = float(value)



SUMMARY:
**Summary:**

The `_convert_str_dict` function takes a dictionary as input, ensuring it conforms to a specific structure. It:
- Recursively converts nested dictionaries
- Converts string values to their corresponding types (bool, int, float) based on the value's format.

**Key Implementation Details:**

* Recursive conversion of nested dictionaries using `self.` method signature.
* Conversion of boolean and numeric values from strings based on specified formats.

**Notable Dependencies or Requirements:**
 
* None.


key: chroma:document
string_value:
CODE:

class ParallelMode(Enum):
    NOT_PARALLEL = "not_parallel"
    NOT_DISTRIBUTED = "not_distributed"
    DISTRIBUTED = "distributed"
    SAGEMAKER_MODEL_PARALLEL = "sagemaker_model_parallel"
    SAGEMAKER_DATA_PARALLEL = "sagemaker_data_parallel"


SUMMARY:
**Class Summary: `ParallelMode` Enum**

The `ParallelMode` class is an enumeration that defines different parallelization modes for machine learning. It provides predefined values to represent distinct scenarios, such as:

* Non-distributed processing
* Distributed data and model parallelism (Sagemaker-specific)
* TPU utilization

**Key Implementation Details:**

* Four classes define the main parallelization modes:
	+ `NOT_PARALLEL` (string "not_parallel")
	+ `NOT_DISTRIBUTED` (string "not_distributed")
	+ `DISTRIBUTED` (string "distributed")
	+ `SAGEMAKER_MODEL_PARALLEL` and `SAGEMAKER_DATA_PARALLEL` (Sagemaker-specific modes)
* Three additional constants represent TPU utilization:
	+ `TPU` (string "tpu")

**Notable Dependencies/Requirements:**

* None explicitly listed, but may interact with Sagemaker or TPU-related dependencies.


key: chroma:document
string_value:
CODE:

def seed_worker(_):
    """
    Helper function to set worker seed during Dataloader initialization.
    """
    worker_seed = torch.initial_seed() % 2**32


SUMMARY:
**Summary:**

### Function Purpose and Functionality:

`seed_worker` sets the initial seed for PyTorch's random number generator (`torch`) to ensure reproducibility in Dataloader initialization.

### Key Implementation Details:

* Calculates an integer value between 0 and 2^32 using `torch.initial_seed()`.
* Assigns this value to a worker seed using `set_seed(worker_seed)`.

### Notable Dependencies and Requirements:

* PyTorch (`torch`)
* `set_seed` function (part of PyTorch)
* Reproducibility in Dataloader initialization


key: chroma:document
string_value:
CODE:

def enable_full_determinism(seed: int, warn_only: bool = False):
    """
    Helper function for reproducible behavior during distributed training. See
    - https://pytorch.org/docs/stable/notes/randomness.html for pytorch
    - https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism for tensorflow
    """
    # set seed first
    set_seed(seed)

    if is_torch_available():
        # Enable PyTorch deterministic mode. This potentially requires either the environment
        # variable 'CUDA_LAUNCH_BLOCKING' or 'CUBLAS_WORKSPACE_CONFIG' to be set,
        # depending on the CUDA version, so we set them both here
        os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
        os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":16:8"
        # The environment variable required to enable deterministic mode on Ascend NPUs.
        os.environ["ASCEND_LAUNCH_BLOCKING"] = "1"
        os.environ["HCCL_DETERMINISTIC"] = "1"

        os.environ["FLASH_ATTENTION_DETERMINISTIC"] = "1"
        torch.use_deterministic_algorithms(True, warn_only=warn_only)

        # Enable CUDNN deterministic mode
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False

    if is_tf_available():
        import tensorflow as tf



SUMMARY:
Here's a concise summary of the function code chunk:

**`enable_full_determinism` Function**

**Main Purpose and Functionality:**
This function enables full deterministic behavior in PyTorch and/or TensorFlow for reproducible distributed training. It sets environment variables and configuration options to ensure determinism.

**Key Implementation Details:**

* Sets seed using `set_seed(seed)`.
* Enables PyTorch deterministic mode with:
	+ Setting environment variables (`CUDA_LAUNCH_BLOCKING`, `CUBLAS_WORKSPACE_CONFIG`, etc.)
	+ Using `torch.use_deterministic_algorithms(True, warn_only=warn_only)`
* Enabling CUDNN deterministic mode with `torch.backends.cudnn.deterministic = True`.
* In TensorFlow, enables op determinism using `tf.config.experimental.enable_op_determinism()`.

**Notable Dependencies and Requirements:**

* PyTorch (`torch`)
* TensorFlow (with `is_tf_available()` conditional)
* Environment variables such as:
	+ `CUDA_LAUNCH_BLOCKING`
	+ `CUBLAS_WORKSPACE_CONFIG`
	+ `ASCEND_LAUNCH_BLOCKING`
	+ `HCCL_DETERMINISTIC`
	+ `FLASH_ATTENTION_DETERMINISTIC`


key: chroma:document
string_value:
CODE:

def set_seed(seed: int, deterministic: bool = False):
    """
    Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch` and/or `tf` (if installed).

    Args:
        seed (`int`):
            The seed to set.
        deterministic (`bool`, *optional*, defaults to `False`):
            Whether to use deterministic algorithms where available. Can slow down training.
    """
    random.seed(seed)
    np.random.seed(seed)
    if is_torch_available():
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        # ^^ safe to call this function even if cuda is not available
        if deterministic:
            torch.use_deterministic_algorithms(True)
    if is_torch_mlu_available():
        torch.mlu.manual_seed_all(seed)
    if is_torch_musa_available():
        torch.musa.manual_seed_all(seed)
    if is_torch_npu_available():
        torch.npu.manual_seed_all(seed)
    if is_torch_xpu_available():
        torch.xpu.manual_seed_all(seed)
    if is_tf_available():
        import tensorflow as tf

        tf.random.set_seed(seed)
        if deterministic:


SUMMARY:
**Code Summary**

The `set_seed` function (version 0.1.0) sets seeds for various libraries:

*   Imports the necessary functions from `_random_`, `numpy_, `_torch_, and `_tensorflow_`.
*   Sets seeds using:
    *   `random.seed` (`int`)
    *   `np.random.seed` (`int`)
    *   If Torch is installed:

        -   `torch.manual_seed(seed)` (`float`, default=0.0)
            -  Safely called even without CUDA
        -   `torch.cuda.manual_seed_all(seed)` (default None, ignored if `cuda` is not available)
            -  Uses deterministic algorithms when `determinism=True`
        -   `torch.use_deterministic_algorithms(True)` (if available)
*   And checks availability and usage for various Torch versions:
    *   `_Torch MLU_`
    *   `_Torch Musa_
    *   `_Torch NPU_
    *   `_Torch XPU_
*   Sets seeds using TensorFlow's (`int`) and configures its deterministic option.

**Implementation Details**

To ensure reproducibility, this function sets the `seed` for multiple libraries (random, NumPy, Torch, etc.) in a way that prioritizes performance. The use of different techniques and functions may make the code slightly more complex but ultimately provides better coverage to guarantee certain reproducible behavior


key: chroma:document
string_value:
CODE:

def neftune_post_forward_hook(module, input, output):
    """
    Implements the NEFTune forward pass for the model using forward hooks. Note this works only for torch.nn.Embedding
    layers. This method is slightly adapted from the original source code that can be found here:
    https://github.com/neelsjain/NEFTune Simply add it to your model as follows:
    ```python
    model = ...
    model.embed_tokens.neftune_noise_alpha = 0.1
    model.embed_tokens.register_forward_hook(neftune_post_forward_hook)
    ```
    Args:
        module (`torch.nn.Module`):
            The embedding module where the hook is attached. Note that you need to set `module.neftune_noise_alpha` to
            the desired noise alpha value.
        input (`torch.Tensor`):
            The input tensor to the model.
        output (`torch.Tensor`):
            The output tensor of the model (i.e. the embeddings).
    """
    if module.training:
        dims = torch.tensor(output.size(1) * output.size(2))
        mag_norm = module.neftune_noise_alpha / torch.sqrt(dims)
        output = output + torch.zeros_like(output).uniform_(-mag_norm, mag_norm)


SUMMARY:
**Summary**

The `neftune_post_forward_hook` function implements the NEFTune forward pass for PyTorch `torch.nn.Embedding` layers using forward hooks. It adds Gaussian noise to the embeddings by scaling the input tensor.

**Key Implementation Details**

* Applies noise to the output embeddings of a training model.
* Uses the `neftune_noise_alpha` parameter to control the magnitude of added noise.
* Utilizes PyTorch's built-in tensor operations for efficient computation.

**Interfaces and Dependencies**

* Requires PyTorch (`torch`)
* Modifies the `model.neftune_noise_alpha` parameter (settable at runtime)
* Assumed inheritance from PyTorch's embedding module (`module`)

**External Requirements**

 None 

**Notes**

This function implements a simple form of stochastic optimization by adding noise to the output embeddings, which can help improve model generalization.


key: chroma:document
string_value:
CODE:

class EvalPrediction:
    """
    Evaluation output (always contains labels), to be used to compute metrics.

    Parameters:
        predictions (`np.ndarray`): Predictions of the model.
        label_ids (`np.ndarray`): Targets to be matched.
        inputs (`np.ndarray`, *optional*): Input data passed to the model.
        losses (`np.ndarray`, *optional*): Loss values computed during evaluation.
    """

    def __init__(
        self,
        predictions: Union[np.ndarray, Tuple[np.ndarray]],
        label_ids: Union[np.ndarray, Tuple[np.ndarray]],
        inputs: Optional[Union[np.ndarray, Tuple[np.ndarray]]] = None,
        losses: Optional[Union[np.ndarray, Tuple[np.ndarray]]] = None,
    ):
        self.predictions = predictions
        self.label_ids = label_ids
        self.inputs = inputs
        self.losses = losses
        self.elements = (self.predictions, self.label_ids)
        if self.inputs is not None:
            self.elements += (self.inputs,)
        if self.losses is not None:
            self.elements += (self.losses,)

    def __iter__(self):
        return iter(self.elements)

    def __getitem__(self, idx):
        if idx < 0 or idx >= len(self.elements):
            raise IndexError("tuple index out of range")


SUMMARY:
**EvalPrediction Class Summary**

**Main Purpose and Functionality:**
The `EvalPrediction` class represents evaluation output data, containing predictions, labels, inputs, losses, and their associated metadata. It provides methods for iteration and indexing.

**Key Implementation Details:**

* Combines predictions, label IDs, and optional input and loss data into a single container (`elements`)
* Exposes prediction and label ID pairs through iteration and indexing
* Handles optional input and loss data by appending them to the `elements` container

**Notable Dependencies or Requirements:**

* Imports NumPy (`np.ndarray`) as the primary data type
* Uses Python's built-in `Union`, `Optional`, and `Tuple` types for type hinting and flexibility

**Code Quality Notes:**
The class design is straightforward, with clear parameters and a concise implementation. The use of iterators and indexing provides efficient access to the evaluation data. However, consider adding more descriptive docstrings or comments to explain the purpose of each optional parameter (`inputs` and `losses`).


key: chroma:document
string_value:
CODE:

class EvalLoopOutput(NamedTuple):
    predictions: Union[np.ndarray, Tuple[np.ndarray]]
    label_ids: Optional[Union[np.ndarray, Tuple[np.ndarray]]]
    metrics: Optional[Dict[str, float]]


SUMMARY:
**Class Summary: EvalLoopOutput**

The `EvalLoopOutput` class represents the output of an evaluation loop in machine learning models. It contains the following key parameters:

*   **Predictions**: The predicted values, either as a numpy array or a tuple of arrays.
*   **Label IDs**: Optional label information, represented as a numpy array or a tuple of arrays.
*   **Metrics**: A dictionary containing performance metrics with floating-point values (optional).
*   **Num Samples**: The number of samples used for evaluation (optional).

This class is designed to hold the output results of an evaluation loop, which would typically be generated during model training or inference.


key: chroma:document
string_value:
CODE:

class PredictionOutput(NamedTuple):
    predictions: Union[np.ndarray, Tuple[np.ndarray]]
    label_ids: Optional[Union[np.ndarray, Tuple[np.ndarray]]]


SUMMARY:
**Summary:**

The `PredictionOutput` class represents the output of a prediction model. It is a `NamedTuple` with three attributes:
- `predictions`: A Union type containing either a numpy array or a tuple of numpy arrays representing the predicted values.
- `label_ids`: An Optional Union type containing either a numpy array or a tuple of numpy arrays representing the true labels, or null if not provided.
- `metrics`: An Optional dictionary containing metric scores (float values), possibly empty.

**Notable Implementation Details:**

* Uses NumPy arrays/tuples for predictions, labels, and metrics to ensure data type consistency.
* Employs Python's built-in support for NamedTuples to efficiently represent the class structure.

**Dependencies/Requirements:**

* `np.ndarray` and `Tuple` from NumPy and/or typing libraries.
* Python 3.6+ support due to `NamedTuple`.


key: chroma:document
string_value:
CODE:

class TrainOutput(NamedTuple):
    global_step: int
    training_loss: float


SUMMARY:
**.train_output.class**

* **Purpose**: Represents the output of a training iteration.
* **Primary Purpose**: To encapsulate key metrics and loss during training.
* **Implementation Details**:
	+ Uses `NamedTuple` for simplicity and type safety.
	+ Includes `global_step`, `training_loss`, and `metrics`.
	+ Note: No method calls or operations are defined; primarily used to hold data.

No external dependencies are referenced, but it assumes the existence of a `Dict` implementation.


key: chroma:document
string_value:
CODE:

def get_last_checkpoint(folder):
    content = os.listdir(folder)
    checkpoints = [
        path
        for path in content
        if _re_checkpoint.search(path) is not None and os.path.isdir(os.path.join(folder, path))
    ]
    if len(checkpoints) == 0:
        return


SUMMARY:
**Summary**

This function (`get_last_checkpoint`) retrieves the most recent checkpoint file from a given folder. It:

*   Iterates through files in the specified folder
*   Filters results using a regular expression match for checkpoints ( `_re_checkpoint`)
*   Returns the path to the newest checked point file if found; otherwise, returns `None`

Key Implementation Details:
- Uses `os.path.isdir` and `int(_re_checkpoint.search(path).groups()[0])` for filtering checkpoint files
- Applies `max` function with a custom sorting criteria using `lambda` expression

Notable Dependencies/Requirements:
- `_re_checkpoint`: Regular Expression Pattern used to match checkpoints (exact pattern not shown in this snippet)
- `os`: System Library providing file system functionality (`listdir`, `path.join`)
- External dependency: `pathlib` or `shutil` libraries are recommended for working with paths, if not using the built-in `os` library


key: chroma:document
string_value:
CODE:

class IntervalStrategy(ExplicitEnum):
    NO = "no"
    STEPS = "steps"


SUMMARY:
**Summary:**

The `IntervalStrategy` class defines an enumeration with three values:

- `NO`: No interval strategy
- `STEPS`: Interval strategy based on steps
- `EPOCH`: Interval strategy based on epochs

No key implementation details or dependencies are provided in this code snippet. The explicit enumeration is likely used to represent different strategies for updating model properties within the class.

**RAG System Code Summary:**

* **Purpose:** Enumerates possible interval strategies.
* **Implementation Details:** Not applicable in this excerpt.
* **Notable Interactions:** None.
* **External Dependencies:** N/A.


key: chroma:document
string_value:
CODE:

class SaveStrategy(ExplicitEnum):
    NO = "no"
    STEPS = "steps"
    EPOCH = "epoch"


SUMMARY:
**SaveStrategy Enum Summary**

The `SaveStrategy` enum defines four possible strategies for saving model checkpoints:

- **NO**: Do not save models.
- **STEPS**: Save models at each training step.
- **EPOCH**: Save models only after completing an entire epoch.
- **BEST**: Save the best-performing model so far.

No key implementation details or notable dependencies are required for this simple enum.


key: chroma:document
string_value:
CODE:

class EvaluationStrategy(ExplicitEnum):
    NO = "no"
    STEPS = "steps"


SUMMARY:
**Summary:**

The `EvaluationStrategy` class defines an enumeration for evaluating machine learning models. 

*   **Purpose:** Specifies evaluation strategies for model performance assessment.
*   **Implementation Details:**
    *   Enum values: `NO` (No evaluation), `STEPS` (Evaluate after each step), and `EPOCH` (Evaluate at the end of each epoch).
*   **Dependencies: None**


key: chroma:document
string_value:
CODE:

class HubStrategy(ExplicitEnum):
    END = "end"
    EVERY_SAVE = "every_save"
    CHECKPOINT = "checkpoint"


SUMMARY:
**Summary:**

`HubStrategy` Class:

The `HubStrategy` class is an enumeration that represents different strategies for managing data and checkpoints in an `ExplicitEnum`. It provides four options: `END`, `EVERY_SAVE`, `CHECKPOINT`, and `ALL_CHECKPOINTS`.

**Key Details:**

* Four distinct strategy values using string literals (`str`) enums.
* No implementation details except enumeration itself.

**Notable Dependencies:**

None, this is a standalone class definition.


key: chroma:document
string_value:
CODE:

class BestRun(NamedTuple):
    """
    The best run found by a hyperparameter search (see [`~Trainer.hyperparameter_search`]).

    Parameters:
        run_id (`str`):
            The id of the best run (if models were saved, the corresponding checkpoint will be in the folder ending
            with run-{run_id}).
        objective (`float`):
            The objective that was obtained for this run.
        hyperparameters (`Dict[str, Any]`):
            The hyperparameters picked to get this run.
        run_summary (`Optional[Any]`):
            A summary of tuning experiments. `ray.tune.ExperimentAnalysis` object for Ray backend.
    """

    run_id: str
    objective: Union[float, List[float]]
    hyperparameters: Dict[str, Any]


SUMMARY:
**Class Summary: `BestRun`**

*   **Main Purpose:** Represents the best run found by a hyperparameter search in a machine learning model.
*   **Key Implementation Details:**
    *   Stores the ID, objective value, and hyperparameters of the best run.
*   **Notable Dependencies:**
    *   `Trainer.hyperparameter_search`: The method used to perform the hyperparameter search that led to this best run.
*   **Interfaces:** None
*   **External Dependencies:** Ray Tune (specifically, `ray.tune.ExperimentAnalysis`)

This class is designed to store and provide information about the best-performing model found through a hyperparameter tuning process.


key: chroma:document
string_value:
CODE:

def default_compute_objective(metrics: Dict[str, float]) -> float:
    """
    The default objective to maximize/minimize when doing an hyperparameter search. It is the evaluation loss if no
    metrics are provided to the [`Trainer`], the sum of all metrics otherwise.

    Args:
        metrics (`Dict[str, float]`): The metrics returned by the evaluate method.

    Return:
        `float`: The objective to minimize or maximize
    """
    metrics = copy.deepcopy(metrics)
    loss = metrics.pop("eval_loss", None)
    _ = metrics.pop("epoch", None)
    # Remove speed metrics
    speed_metrics = [
        m
        for m in metrics.keys()
        if m.endswith("_runtime") or m.endswith("_per_second") or m.endswith("_compilation_time")
    ]
    for sm in speed_metrics:
        _ = metrics.pop(sm, None)


SUMMARY:
**Summary:**

The `default_compute_objective` function calculates the evaluation objective to minimize or maximize during hyperparameter search. It takes a dictionary of metrics returned by the `Trainer` and returns:

* The evaluation loss if no metrics are provided
* The sum of all metrics if available, excluding speed metrics (e.g., "_runtime", "_per_second", "_compilation_time")

**Key Details:**

* Takes a `Dict[str, float]` input `metrics`
* Returns a single `float` value
* Uses `copy.deepcopy(metrics)` to create a copy of the input dictionary
* Removes "eval_loss" and "epoch" keys from the original dictionary
* Pops speed metrics using a list comprehension
* Returns the loss if all non-speed metrics are removed, otherwise returns the sum of all remaining values

**Notable Dependencies/Requirements:**

* The `copy` module for deep copying the input dictionary
* The `Trainer` class and its `evaluate` method (not shown in this snippet)


key: chroma:document
string_value:
CODE:

def default_hp_space_optuna(trial) -> Dict[str, float]:
    from .integrations import is_optuna_available

    assert is_optuna_available(), "This function needs Optuna installed: `pip install optuna`"
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 1, 5),
        "seed": trial.suggest_int("seed", 1, 40),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [4, 8, 16, 32, 64]),


SUMMARY:
**Function Summary: `default_hp_space_optuna`**

**Primary Purpose:** Generates a hyperparameter search space for Optimizing Hyperparameters using OptUNA.

### Key Implementation Details:

* Uses built-in OptUNA functionality to generate initial hyperparameters:
	+ `learning_rate`: Suggests a float value in the range [1e-6, 1e-4] with log scaling.
	+ `num_train_epochs`: Suggests an integer value between 1 and 5.
	+ `seed`: Suggests an integer value between 1 and 40.
	+ `per_device_train_batch_size`: Suggests a categorical choice from [4, 8, 16, 32, 64].
* Requires OptUNA to be installed (pip install optuna).

### Notable Dependencies/Requirements:

* `.integrations` module for the `is_optuna_available()` function.
* OptUNA library.


key: chroma:document
string_value:
CODE:

def default_hp_space_ray(trial) -> Dict[str, float]:
    from .integrations import is_ray_tune_available

    assert is_ray_tune_available(), "This function needs ray installed: `pip install ray[tune]`"
    from ray import tune

    return {
        "learning_rate": tune.loguniform(1e-6, 1e-4),
        "num_train_epochs": tune.choice(list(range(1, 6))),
        "seed": tune.uniform(1, 40),
        "per_device_train_batch_size": tune.choice([4, 8, 16, 32, 64]),


SUMMARY:
**Summary:**

This function generates a default hyperparameter space for Ray Tune's `tune` module. 

It searches the specified hyperparameters with the following distributions:

- `learning_rate`: Loguniform between 10^-6 and 10^-4
- `num_train_epochs`: Choice of integer values from 1 to 5
- `seed`: Uniform value between 1 and 40
- `per_device_train_batch_size`: Choice of integer values [4, 8, 16, 32, 64]

The function also ensures Ray Tune is installed by checking for its availability. 

**Notable dependencies:**
- `ray` (with `tune` module)
- `.integrations.is_ray_tune_available()` (utility function)


key: chroma:document
string_value:
CODE:

def default_hp_space_sigopt(trial):
    return [
        {"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double", "transformamtion": "log"},
        {"bounds": {"min": 1, "max": 6}, "name": "num_train_epochs", "type": "int"},
        {"bounds": {"min": 1, "max": 40}, "name": "seed", "type": "int"},
        {
            "categorical_values": ["4", "8", "16", "32", "64"],
            "name": "per_device_train_batch_size",
            "type": "categorical",
        },


SUMMARY:
**Summary**

This function generates a Hyperparameter Search Space for SigOpt, a Bayesian optimization algorithm. It defines four hyperparameters:

* `learning_rate`: A logarithmically transformed double variable between 1e-6 and 1e-4 bounds.
* `num_train_epochs`: An integer variable between 1 and 6.
* `seed`: An integer variable between 1 and 40.
* `per_device_train_batch_size`: A categorical variable with discrete values [64, 32, 16, 8, 4].

Note: The `transformamtion` field (typo: should be "transformation") is likely a formatting error and is not part of the intended hyperparameter search space.

**Key Details**

* Uses SigOpt's `categorical_values` for `per_device_train_batch_size`.
* Applies logarithmic transformation to `learning_rate`.

**Dependencies**

* None listed.
* Assumed use of the `SigOpt` library, which is not explicitly imported in this snippet.


key: chroma:document
string_value:
CODE:

def default_hp_space_wandb(trial) -> Dict[str, float]:
    from .integrations import is_wandb_available

    if not is_wandb_available():
        raise ImportError("This function needs wandb installed: `pip install wandb`")

    return {
        "method": "random",
        "metric": {"name": "objective", "goal": "minimize"},
        "parameters": {
            "learning_rate": {"distribution": "uniform", "min": 1e-6, "max": 1e-4},
            "num_train_epochs": {"distribution": "int_uniform", "min": 1, "max": 6},
            "seed": {"distribution": "int_uniform", "min": 1, "max": 40},
            "per_device_train_batch_size": {"values": [4, 8, 16, 32, 64]},
        },


SUMMARY:
**Summary**

`default_hp_space_wandb` function generates a hyperparameter search space using WandB. The primary purpose is to define a configuration for hyperparameter tuning.

**Key Implementation Details:**

*   **WandB Initialization**: Checks if `wandb` is installed and raises an error if not.
*   **Hyperparameter Search Space**: Defines a dictionary with three main components:
    *   `method`: Random search.
    *   `metric`: Minimize the "objective" metric.
    *   `parameters`: Dict containing tuned hyperparameters with distribution constraints.

**Notable Dependencies and Requirements:**

*   `wandb` (WandB library)

**Summary Format:** `Default HP Space WandB` uses `random` search with `uniform` distributions for some parameters, while others are constrained to specific ranges or discrete values.


key: chroma:document
string_value:
CODE:

class HPSearchBackend(ExplicitEnum):
    OPTUNA = "optuna"
    RAY = "ray"
    SIGOPT = "sigopt"


SUMMARY:
**Class Summary: HPSearchBackend**

The `HPSearchBackend` class defines an enumeration for Hyperparameter Search backend options.

**Main Purpose and Functionality**
Provides a list of supported backend options for hyperparameter search, including Optuna, Ray, SigOpt, and WandB.

**Key Implementation Details**
None notable.

**Notable Dependencies or Requirements**

* Enum values: `Optuna`, `Ray`, `SigOpt`, `WandB`
* External dependencies: None specified but likely references to these backends.

This summary provides a concise overview of the `HPSearchBackend` class, highlighting its main purpose and key implementation details.


key: chroma:document
string_value:
CODE:

def is_main_process(local_rank):
    """
    Whether or not the current process is the local process, based on `xm.get_ordinal()` (for TPUs) first, then on
    `local_rank`.
    """
    if is_torch_xla_available():
        import torch_xla.core.xla_model as xm

        return xm.get_ordinal() == 0


SUMMARY:
**Function Summary: `is_main_process`**

This function determines whether the current process is the main process. It first checks if Torch XLA is available, and if so, uses its `get_ordinal()` method to determine the main process ID ( defaults to 0 for TPUs). If not, it returns true if the local rank (`local_rank`) is -1 or 0.

**Key Implementation Details:**

* `xm.get_ordinal()` is used to get the ordinal ID of the process.
* `is_torch_xla_available()` is called before utilizing Torch XLA functionality.

**Notable Dependencies and Requirements:**

* None, as this function only uses built-in Python functionality.
* External dependency on: Torch XLA (imported module)


key: chroma:document
string_value:
CODE:

def total_processes_number(local_rank):
    """
    Return the number of processes launched in parallel. Works with `torch.distributed` and TPUs.
    """
    if is_torch_xla_available():
        import torch_xla.core.xla_model as xm

        return xm.xrt_world_size()
    elif local_rank != -1 and is_torch_available():
        import torch

        return torch.distributed.get_world_size()


SUMMARY:
**Summary:**

The `total_processes_number` function determines the total number of processes launched in parallel, supporting multiple frameworks (`torch.xla`, `torch`) and hardware platforms (TPUs). It returns:

* The number of XLA-enabled processes using `torch_xla.core.xla_model`
* The number of tensor processing units (TPUs) enabled for GPU acceleration by `torch.distributed` when `local_rank != -1` and `torch` is available
* 1 as the default value in case none of the above are supported

**Key aspects:**

* Supports multiple frameworks (`torch.xla`, `torch`)
* Handles TPU hardware acceleration with `torch.distributed`
* Defaults to 1 if no supported setup is found


key: chroma:document
string_value:
CODE:

def speed_metrics(split, start_time, num_samples=None, num_steps=None, num_tokens=None):
    """
    Measure and return speed performance metrics.

    This function requires a time snapshot `start_time` before the operation to be measured starts and this function
    should be run immediately after the operation to be measured has completed.

    Args:
    - split: name to prefix metric (like train, eval, test...)
    - start_time: operation start time
    - num_samples: number of samples processed
    - num_steps: number of steps processed
    - num_tokens: number of tokens processed
    """
    runtime = time.time() - start_time
    result = {f"{split}_runtime": round(runtime, 4)}
    if runtime == 0:
        return result
    if num_samples is not None:
        samples_per_second = num_samples / runtime
        result[f"{split}_samples_per_second"] = round(samples_per_second, 3)
    if num_steps is not None:
        steps_per_second = num_steps / runtime
        result[f"{split}_steps_per_second"] = round(steps_per_second, 3)
    if num_tokens is not None:
        tokens_per_second = num_tokens / runtime
        result[f"{split}_tokens_per_second"] = round(tokens_per_second, 3)


SUMMARY:
**Function Summary:**

`speed_metrics`

Measures and returns speed performance metrics for a given operation.

**Primary Purpose:**
Calculate execution time and throughput of an operation in terms of samples, steps, and tokens processed per second.

**Key Implementation Details:**

* Calculates runtime using `time.time() - start_time`.
* Computes and stores metrics (samples_per_second, steps_per_second, tokens_per_second) if input values are provided.
* Returns a dictionary containing metric results with prefixed string keys.

**Notable Dependencies/Requirements:**

* Uses `time` module for timestamp calculations.
* Does not impose any external interface requirements or dependencies explicitly stated in the provided code chunk.


key: chroma:document
string_value:
CODE:

class SchedulerType(ExplicitEnum):
    """
    Scheduler names for the parameter `lr_scheduler_type` in [`TrainingArguments`].
    By default, it uses "linear". Internally, this retrieves `get_linear_schedule_with_warmup` scheduler from [`Trainer`].
    Scheduler types:
       - "linear" = get_linear_schedule_with_warmup
       - "cosine" = get_cosine_schedule_with_warmup
       - "cosine_with_restarts" = get_cosine_with_hard_restarts_schedule_with_warmup
       - "polynomial" = get_polynomial_decay_schedule_with_warmup
       - "constant" =  get_constant_schedule
       - "constant_with_warmup" = get_constant_schedule_with_warmup
       - "inverse_sqrt" = get_inverse_sqrt_schedule
       - "reduce_lr_on_plateau" = get_reduce_on_plateau_schedule
       - "cosine_with_min_lr" = get_cosine_with_min_lr_schedule_with_warmup
       - "warmup_stable_decay" = get_wsd_schedule
    """

    LINEAR = "linear"
    COSINE = "cosine"
    COSINE_WITH_RESTARTS = "cosine_with_restarts"
    POLYNOMIAL = "polynomial"
    CONSTANT = "constant"
    CONSTANT_WITH_WARMUP = "constant_with_warmup"
    INVERSE_SQRT = "inverse_sqrt"
    REDUCE_ON_PLATEAU = "reduce_lr_on_plateau"
    COSINE_WITH_MIN_LR = "cosine_with_min_lr"


SUMMARY:
**Summary**

The `SchedulerType` class defines an enumeration of scheduler names used in training, mapping to specific variants (e.g., linear, cosine) that are implemented by corresponding methods (`get_linear_schedule_with_warmup`, etc.). The class provides a way to categorize and retrieve these scheduling strategies.

**Key Implementation Details**

* Uses `Trainer` internally to retrieve specific schedules
* Implements the following scheduling models:
	+ Linear: `get_linear_schedule_with_warmup`
	+ Cosine: `get_cosine_schedule_with_warmup`
	+ Cosine with restarts: `get_cosine_with_hard_restarts_schedule_with_warmup`
	+ Polynomial decay: `get_polynomial_decay_schedule_with_warmup`
	+ Constant schedule: `get_constant_schedule`
	+ Constant with warm-up: `get_constant_schedule_with_warmup`
	+ Inverse sqrt schedule: `get_inverse_sqrt_schedule`
	+ Reduce on plateau schedule: `get_reduce_on_plateau_schedule`
	+ Cosine with min LR: `get_cosine_with_min_lr_schedule_with_warmup`
	+ Warm-up stable decay: `get_wsd_schedule`

**Notable Dependencies**

* `ExplicitEnum` used for class definition
* `TrainingArguments` referenced in documentation
* `Trainer` used internally to retrieve schedules


key: chroma:document
string_value:
CODE:

class TrainerMemoryTracker:
    """
    A helper class that tracks cpu and gpu memory.

    This class will silently skip unless `psutil` is available. Install with `pip install psutil`.

    When a stage completes, it can pass metrics dict to update with the memory metrics gathered during this stage.

    Example :

    ```python
    self._memory_tracker = TrainerMemoryTracker(self.args.skip_memory_metrics)
    self._memory_tracker.start()
    # code ...
    metrics = {"train_runtime": 10.5}
    self._memory_tracker.stop_and_update_metrics(metrics)
    ```

    At the moment GPU tracking is only for `pytorch`, but can be extended to support `tensorflow`.

    To understand this class' intricacies please read the documentation of [`~Trainer.log_metrics`].
    """

    # map trainer methods to metrics prefix
    stages = {
        "__init__": "init",
        "train": "train",
        "_inner_training_loop": "train",
        "evaluate": "eval",
        "predict": "test",
    }

    def __init__(self, skip_memory_metrics=False):
        self.skip_memory_metrics = skip_memory_metrics

        if not is_psutil_available():
            # soft dependency on psutil
            self.skip_memory_metrics = True

        if self.skip_memory_metrics:
            return

        import psutil  # noqa

        if is_torch_cuda_available() or is_torch_mlu_available() or is_torch_musa_available():
            import torch

            self.torch = torch
            self.gpu = {}
        elif is_torch_mps_available():
            import torch

            self.torch = torch
            self.gpu = {}
        elif is_torch_xpu_available():
            import torch

            self.torch = torch
            self.gpu = {}
        elif is_torch_npu_available():
            import torch

            self.torch = torch
            self.gpu = {}
        else:
            self.torch = None

        self.process = psutil.Process()

        self.cur_stage = None
        self.cpu = {}
        self.init_reported = False

    def derive_stage(self):
        """derives the stage/caller name automatically"""
        caller = inspect.currentframe().f_back.f_back.f_code.co_name
        if caller in self.stages:
            return self.stages[caller]
        else:
            raise ValueError(
                f"was called from {caller}, but only expect to be called from one of {self.stages.keys()}"
            )

    def cpu_mem_used(self):
        """get resident set size memory for the current process"""
        return self.process.memory_info().rss

    def peak_monitor_func(self):
        self.cpu_mem_used_peak = -1

        while True:
            self.cpu_mem_used_peak = max(self.cpu_mem_used(), self.cpu_mem_used_peak)

            # can't sleep or will not catch the peak right (this comment is here on purpose)
            # time.sleep(0.001) # 1msec

            if not self.peak_monitoring:
                break

    def start(self):
        """start tracking for the caller's stage"""
        if self.skip_memory_metrics:
            return

        stage = self.derive_stage()
        # deal with nested calls of eval during train - simply ignore those
        if self.cur_stage is not None and self.cur_stage != stage:
            return

        self.cur_stage = stage

        gc.collect()

        if self.torch is not None:
            if torch.cuda.is_available():
                self.torch.cuda.reset_peak_memory_stats()
                self.torch.cuda.empty_cache()
            elif is_torch_mlu_available():
                self.torch.mlu.reset_peak_memory_stats()
                self.torch.mlu.empty_cache()
            elif is_torch_musa_available():
                self.torch.musa.reset_peak_memory_stats()
                self.torch.musa.empty_cache()
            elif is_torch_xpu_available():
                self.torch.xpu.reset_peak_memory_stats()
                self.torch.xpu.empty_cache()
            elif is_torch_npu_available():
                self.torch.npu.reset_peak_memory_stats()
                self.torch.npu.empty_cache()
            elif is_torch_mps_available():
                self.torch.mps.empty_cache()

        # gpu
        if self.torch is not None:
            if torch.cuda.is_available():
                self.gpu_mem_used_at_start = self.torch.cuda.memory_allocated()
            elif is_torch_mlu_available():
                self.gpu_mem_used_at_start = self.torch.mlu.memory_allocated()
            elif is_torch_musa_available():
                self.gpu_mem_used_at_start = self.torch.musa.memory_allocated()
            elif is_torch_xpu_available():
                self.gpu_mem_used_at_start = self.torch.xpu.memory_allocated()
            elif is_torch_npu_available():
                self.gpu_mem_used_at_start = self.torch.npu.memory_allocated()
            elif is_torch_mps_available():
                self.gpu_mem_used_at_start = self.torch.mps.current_allocated_memory()

        # cpu
        self.cpu_mem_used_at_start = self.cpu_mem_used()

        self.peak_monitoring = True
        peak_monitor_thread = threading.Thread(target=self.peak_monitor_func)
        peak_monitor_thread.daemon = True
        peak_monitor_thread.start()

    def stop(self, stage):
        """stop tracking for the passed stage"""

        # deal with nested calls of eval during train - simply ignore those
        if self.cur_stage is not None and self.cur_stage != stage:
            return

        # this sends a signal to peak_monitor_func to complete its loop
        self.peak_monitoring = False

        # first ensure all objects get collected and their memory is freed
        gc.collect()

        if self.torch is not None:
            if torch.cuda.is_available():
                self.torch.cuda.empty_cache()
            elif is_torch_mlu_available():
                self.torch.mlu.empty_cache()
            elif is_torch_musa_available():
                self.torch.musa.empty_cache()
            elif is_torch_xpu_available():
                self.torch.xpu.empty_cache()
            elif is_torch_npu_available():
                self.torch.npu.empty_cache()
            elif is_torch_mps_available():
                self.torch.mps.empty_cache()

        # concepts:
        # - alloc_delta:  the difference of allocated memory between the end and the start
        # - peaked_delta: the difference between the peak memory and the current memory
        # in order to know how much memory the measured code consumed one needs to sum these two

        # gpu
        if self.torch is not None:
            if torch.cuda.is_available():
                self.gpu_mem_used_now = self.torch.cuda.memory_allocated()
                self.gpu_mem_used_peak = self.torch.cuda.max_memory_allocated()
            elif is_torch_mlu_available():
                self.gpu_mem_used_now = self.torch.mlu.memory_allocated()
                self.gpu_mem_used_peak = self.torch.mlu.max_memory_allocated()
            elif is_torch_musa_available():
                self.gpu_mem_used_now = self.torch.musa.memory_allocated()
                self.gpu_mem_used_peak = self.torch.musa.max_memory_allocated()
            elif is_torch_xpu_available():
                self.gpu_mem_used_now = self.torch.xpu.memory_allocated()
                self.gpu_mem_used_peak = self.torch.xpu.max_memory_allocated()
            elif is_torch_npu_available():
                self.gpu_mem_used_now = self.torch.npu.memory_allocated()
                self.gpu_mem_used_peak = self.torch.npu.max_memory_allocated()
            elif is_torch_mps_available():
                self.gpu_mem_used_now = self.torch.mps.current_allocated_memory()
                # self.torch.mps.max_memory_allocated() does not exist yet
                self.gpu_mem_used_peak = None

            else:
                raise ValueError("No available GPU device found!")

            self.gpu[self.cur_stage] = {
                "begin": self.gpu_mem_used_at_start,
                "end": self.gpu_mem_used_now,
                "alloc": (self.gpu_mem_used_now - self.gpu_mem_used_at_start),
            }
            if self.gpu_mem_used_peak is not None:
                self.gpu[self.cur_stage]["peaked"] = max(0, self.gpu_mem_used_peak - self.gpu_mem_used_now)
            else:
                self.gpu[self.cur_stage]["peaked"] = "Not available"

        # cpu
        self.cpu_mem_used_now = self.cpu_mem_used()
        self.cpu[self.cur_stage] = {
            "begin": self.cpu_mem_used_at_start,
            "end": self.cpu_mem_used_now,
            "alloc": (self.cpu_mem_used_now - self.cpu_mem_used_at_start),
            "peaked": max(0, self.cpu_mem_used_peak - self.cpu_mem_used_now),
        }

        # reset - cycle finished
        self.cur_stage = None

    def update_metrics(self, stage, metrics):
        """updates the metrics"""
        if self.skip_memory_metrics:
            return

        # deal with nested calls of eval during train - simply ignore those
        if self.cur_stage is not None and self.cur_stage != stage:
            return

        # since we don't have a way to return init metrics, we push them into the first of train/val/predict
        stages = [stage]
        if not self.init_reported:
            stages.insert(0, "init")
            self.init_reported = True

        for stage in stages:
            for t in ["alloc", "peaked"]:
                if stage in self.cpu and t in self.cpu[stage]:
                    metrics[f"{stage}_mem_cpu_{t}_delta"] = self.cpu[stage][t]
                if self.torch is not None and stage in self.gpu and t in self.gpu[stage]:
                    metrics[f"{stage}_mem_gpu_{t}_delta"] = self.gpu[stage][t]
            # if we need additional debug info, enable the following
            # for t in ["begin", "end"]:
            #     if stage in self.cpu and t in self.cpu[stage]:
            #         metrics[f"{stage}_mem_cpu_{t}"] = self.cpu[stage][t]
            #     if self.torch is not None and stage in self.gpu and t in self.gpu[stage]:
            #         metrics[f"{stage}_mem_gpu_{t}"] = self.gpu[stage][t]

        # since memory can be allocated before init, and it might be difficult to track overall
        # memory usage, in particular for GPU, let's report memory usage at the point init was called
        if stages[0] == "init":
            metrics["before_init_mem_cpu"] = self.cpu["init"]["begin"]
            if self.torch is not None:
                metrics["before_init_mem_gpu"] = self.gpu["init"]["begin"]
            # if we also wanted to report any additional memory allocations in between init and
            # whatever the next stage was we could also report this:
            # if self.cpu["init"]["end"] != self.cpu[stage]["begin"]:
            #     metrics[f"after_init_mem_cpu_delta"] = self.cpu[stage]["begin"] - self.cpu["init"]["end"]
            # if self.torch is not None and self.gpu["init"]["end"] != self.gpu[stage]["begin"]:
            #     metrics[f"after_init_mem_gpu_delta"] = self.gpu[stage]["begin"] - self.gpu["init"]["end"]

    def stop_and_update_metrics(self, metrics=None):
        """combine stop and metrics update in one call for simpler code"""
        if self.skip_memory_metrics:
            return

        stage = self.derive_stage()
        self.stop(stage)

        # init doesn't have metrics to update so we just save that data for later stages to retrieve
        if metrics is not None:


SUMMARY:
Here's a concise summary of the provided code in the context of memory tracking and metrics updating:

**Memory Tracker Class**

The `MemoryTracker` class is designed to track memory usage for CPU and GPU allocations during the execution of various algorithms (e.g., training, validation, prediction phases). It provides methods to record memory usage at specific stages (`init`, `begin`, `end`) and update internal memory tracking data structures.

**Key Features:**

1. **Tracking Memory Usage**: The class tracks memory usage for CPU and GPU allocations across different stages of algorithm execution.
2. **Metrics Updates**: The class updates metrics (e.g., allocatable memory, peak memory usage) for each stage, allowing users to easily retrieve and compare memory usage throughout the algorithm's lifecycle.
3. **Support for Nested Algorithm Calls**: The class handles nested calls to algorithms by pushing initial metrics into the first training/val/prediction phase and then retrieving them from subsequent stages.

**Example Usage**

```python
tracker = MemoryTracker()  # Initialize a new memory tracker

# During a training loop:
tracker.update_metrics("train", {"metrics": {}})  # Update metrics for this stage

# After completing the loop...
tracker.stop_and_update_metrics({"metrics": {}})  # Combine stop and metrics update, returning final metrics
```

Overall, the `MemoryTracker` class offers an efficient way to track memory usage during complex algorithmic workflows.


key: chroma:document
string_value:
CODE:

def has_length(dataset):
    """
    Checks if the dataset implements __len__() and it doesn't raise an error
    """
    try:
        return len(dataset) is not None
    except TypeError:
        # TypeError: len() of unsized object


SUMMARY:
**Summary**

`has_length` function checks if a dataset implements `__len__()`. Returns `True` if the dataset's length can be determined without raising an error, and `False` otherwise.

**Key Details**

* Uses `try-except` block to handle potential TypeError
* Checks if `dataset` supports `len()` operation
* Returns immediately when successful

**Note**: The function is not explicitly documented for handling scenarios where `dataset` raises a different type of error. It may be beneficial to add additional error handling or clarification for such cases.


key: chroma:document
string_value:
CODE:

def denumpify_detensorize(metrics):
    """
    Recursively calls `.item()` on the element of the dictionary passed
    """
    if isinstance(metrics, (list, tuple)):
        return type(metrics)(denumpify_detensorize(m) for m in metrics)
    elif isinstance(metrics, dict):
        return type(metrics)({k: denumpify_detensorize(v) for k, v in metrics.items()})
    elif isinstance(metrics, np.generic):
        return metrics.item()
    elif is_torch_available() and isinstance(metrics, torch.Tensor) and metrics.numel() == 1:
        return metrics.item()


SUMMARY:
**Summary**

This function (`denumpify_detensorize`) recursively converts a nested dictionary or list containing NumPy numeric types to numerical `float` or integer values, while maintaining structural integrity. It leverages Python's tuple/dict unpacking and `type()` casting to apply the conversion.

**Key Implementation Details**

* Recursively calls `.item()` on each element of lists/tuples/tensors
* Converts dictionaries using `{k: denumpify_detensorize(v) for k, v in metrics.items()}`

**Notable Dependencies/Requirements**

* NumPy (`np.generic`)
* Torch (conditional check `is_torch_available()` and `torch.Tensor`)

This function aims to simplify nested numeric data structures while preserving their original structure.


key: chroma:document
string_value:
CODE:

def number_of_arguments(func):
    """
    Return the number of arguments of the passed function, even if it's a partial function.
    """
    if isinstance(func, functools.partial):
        total_args = len(inspect.signature(func.func).parameters)
        return total_args - len(func.args) - len(func.keywords)


SUMMARY:
**Function Summary**

`number_of_arguments` function calculates the number of arguments for a given function, including partial functions.

**Main Purpose and Functionality:**

Returns the total number of parameters in a function's signature, accounting for partial functions.

**Key Implementation Details:**

* Checks if `func` is a partial function using `functools.partial`.
* If not partial, uses `inspect.signature(func)` to retrieve function signature.
* For partial functions, calculates total arguments by subtracting args and keywords from the function's full signature size.
* Returns the calculated total number of arguments.

**Notable Dependencies or Requirements:**

* `functools` (partial support).
* `inspect` (signature parsing).

**Code Summary:**
```
def number_of_arguments(func):
    """
    Calculate the total number of arguments for a function, including partial functions.
    """
    if isinstance(func, functools.partial):
        # Use full signature size and adjust for args and keywords
        return len(inspect.signature(func.func).parameters) - len(func.args) - len(func.keywords)
    else:
        # Retrieve function signature and count parameters
        return len(inspect.signature(func).parameters)
```


key: chroma:document
string_value:
CODE:

def find_executable_batch_size(
    function: callable = None, starting_batch_size: int = 128, auto_find_batch_size: bool = False
):
    """
    Args:
    A basic decorator that will try to execute `function`. If it fails from exceptions related to out-of-memory or
    CUDNN, the batch size is cut in half and passed to `function`. `function` must take in a `batch_size` parameter as
    its first argument.
        function (`callable`, *optional*)
            A function to wrap
        starting_batch_size (`int`, *optional*)
            The batch size to try and fit into memory
        auto_find_batch_size (`bool`, *optional*)
            If False, will just execute `function`
    """
    if function is None:
        return functools.partial(
            find_executable_batch_size,
            starting_batch_size=starting_batch_size,
            auto_find_batch_size=auto_find_batch_size,
        )

    if auto_find_batch_size:
        requires_backends(find_executable_batch_size, "accelerate")
        from accelerate.utils import find_executable_batch_size as accelerate_find_executable_batch_size

        return accelerate_find_executable_batch_size(function=function, starting_batch_size=starting_batch_size)



SUMMARY:
**Summary: `find_executable_batch_size` Function**

**Purpose:** Dynamically adjusts the batch size for a given executable function to prevent memory or CUDNN errors.

**Implementation Details:**

* The function takes in a callable `function`, an initial `starting_batch_size`, and an optional `auto_find_batch_size` flag.
* If `function` is None, returns a partial function with default arguments.
* If `auto_find_batch_size` is True, uses the `requires_backends` function to check for CUDA acceleration. Otherwise, applies the batch size directly.

**Key Components:**

* `functools.partial` used to create new functions with adjusted arguments
* `accelerate.utils.find_executable_batch_size` used when `auto_find_batch_size` is True

**Notable Dependencies:**

* `accelerate`
* `functools`


key: chroma:document
string_value:
CODE:

class FSDPOption(ExplicitEnum):
    FULL_SHARD = "full_shard"
    SHARD_GRAD_OP = "shard_grad_op"
    NO_SHARD = "no_shard"
    HYBRID_SHARD = "hybrid_shard"
    HYBRID_SHARD_ZERO2 = "hybrid_shard_zero2"
    OFFLOAD = "offload"


SUMMARY:
**Summary**

This class defines an enumeration `FSDPOption` with six predefined values, each representing a specific configuration option for FSDP (Full Stack Distributed Processing). The options include full/shard, grad op, no shard, hybrid shard, offload, and auto wrap.

**Key Highlights:**

* Enumerated values defined for configuration purposes in FSDP
* Exclusive and explicit naming convention (`ExplicitEnum`)
* No key algorithm or design choice implementation details mentioned

**Notable Dependencies:**

* None explicitly referenced

Note: The exact application context of this enumeration is not provided, but based on the options listed, it appears to be related to parallel processing or distributed computing configurations.


key: chroma:document
string_value:
CODE:

class RemoveColumnsCollator:
    """Wrap the data collator to remove unused columns before they are passed to the collator."""

    def __init__(
        self,
        data_collator,
        signature_columns,
        logger=None,
        model_name: Optional[str] = None,
        description: Optional[str] = None,
    ):
        self.data_collator = data_collator
        self.signature_columns = signature_columns
        self.logger = logger
        self.description = description
        self.model_name = model_name
        self.message_logged = False

    def _remove_columns(self, feature: dict) -> dict:
        if not isinstance(feature, dict):
            return feature
        if not self.message_logged and self.logger and self.model_name:
            ignored_columns = list(set(feature.keys()) - set(self.signature_columns))
            if len(ignored_columns) > 0:
                dset_description = "" if self.description is None else f"in the {self.description} set"
                self.logger.info(
                    f"The following columns {dset_description} don't have a corresponding argument in "
                    f"`{self.model_name}.forward` and have been ignored: {', '.join(ignored_columns)}."
                    f" If {', '.join(ignored_columns)} are not expected by `{self.model_name}.forward`, "
                    " you can safely ignore this message."
                )
                self.message_logged = True
        return {k: v for k, v in feature.items() if k in self.signature_columns}

    def __call__(self, features: List[dict]):
        features = [self._remove_columns(feature) for feature in features]


SUMMARY:
**RemoveColumnsCollator Class Summary**

The `RemoveColumnsCollator` class wraps a data collator to remove unused columns from input data before passing it to the underlying collator.

**Main Purpose and Functionality:**

- Removes columns not specified in `signature_columns` from input data.
- Logs warnings for ignored columns if `logger` and `model_name` are provided.

**Key Implementation Details:**

- `_remove_columns` function filters out unused columns based on `signature_columns`.
- Uses list comprehension to efficiently create filtered features.
- Logs information message using logger if necessary.

**Notable Dependencies or Requirements:**

* `data_collator`: Wrapped data collator class.
* `signature_columns`: List of valid column names for input data.

**Example Method Signature:**
```python
def __call__(self, features: List[dict]) -> List[Dict[str, Any]]:
    ...
```
This code summary provides a concise overview of the `RemoveColumnsCollator` class, highlighting its main purpose, key implementation details, and notable dependencies.


key: chroma:document
string_value:
CODE:

def check_target_module_exists(optim_target_modules, key: str, return_is_regex: bool = False):
    """A helper method to check if the passed module's key name matches any of the target modules in the optim_target_modules.

    Args:
        optim_target_modules (`Union[str, List[str]]`):
            A list of strings to try to match. Can be also a full string.
        key (`str`):
            A key to search any matches in optim_target_modules
        return_is_regex (`bool`):
            If set to `True`, the method will return whether the passed `optim_target_modules`
            is a regex or not.

    Returns:
        `bool` : True of match object if key matches any target modules from config, False or
        None if no match found
        `bool` : If the matched target module is a regex to silence out the warnings in Trainer
        for extra modules being found (only if `target_module_found=True` for an array of regex).
    """
    target_module_found = False
    is_regex = False

    if isinstance(optim_target_modules, str):
        target_module_found = bool(re.fullmatch(optim_target_modules, key))
        is_regex = True if not optim_target_modules == key else False
    elif key in optim_target_modules:  # from here, target_module_found must be a list of str
        # this module is specified directly in target_modules
        target_module_found = True
    elif any(target_key in key for target_key in optim_target_modules):
        target_module_found = True
    elif any(bool(re.fullmatch(optim_target_module, key)) for optim_target_module in optim_target_modules):
        target_module_found = True
        is_regex = True

    if return_is_regex:
        return target_module_found, is_regex



SUMMARY:
**Function Summary**

`check_target_module_exists`: Checks whether a given `key` matches any of the `optim_target_modules`. Returns `True` for match, and optional type information (`is_regex`) if requested.

**Key Implementation Details**

- Uses `re.fullmatch` to check regex patterns
- Allows direct module specification, regex pattern matching, or partial string matching

**Notable Dependencies**

- `Union[str, List[str]]`: Type hint for `optim_target_modules`
- `bool(re.fullmatch())`: `is_regex` calculation
- `any()` and `re.fullmatch()`: Multiple match checks for regex patterns


key: chroma:document
string_value:
CODE:

def import_protobuf_decode_error(error_message=""):
    if is_protobuf_available():
        from google.protobuf.message import DecodeError

        return DecodeError
    else:


SUMMARY:
**Function Summary**

`import_protobuf_decode_error(error_message)` imports `google.protobuf.message.DecodeError` if protobuf is available, otherwise raises an `ImportError`.

**Key Implementation Details**

* Imports `DecodeError` if protobuf is supported (`is_protobuf_available()` returns True).
* Raises custom `ImportError` with formatted error message if protobuf is not available.

**Notable Dependencies/Requirements**

* `google.protobuf.message.DecodeError`
* `is_protobuf_available()`
* Custom `PROTOBUF_IMPORT_ERROR` format string.


key: chroma:document
string_value:
CODE:

class TruncationStrategy(ExplicitEnum):
    """
    Possible values for the `truncation` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in
    an IDE.
    """

    ONLY_FIRST = "only_first"
    ONLY_SECOND = "only_second"
    LONGEST_FIRST = "longest_first"


SUMMARY:
**Concrete Class Summary**

`TruncationStrategy` (Explicit Enum):
   
   Enum class defining possible truncation strategies for tokenization. Its primary purpose is to provide values for the `truncation` argument in the `PreTrainedTokenizerBase.__call__` method.

### Key Implementation Details

* Defines four distinct truncation strategies:
    * `ONLY_FIRST`
    * `ONLY_SECOND`
    * `LONGEST_FIRST`
    * `DO_NOT_TRUNCATE`

### Notable Dependencies or Requirements

None mentioned.
```


key: chroma:document
string_value:
CODE:

class CharSpan(NamedTuple):
    """
    Character span in the original string.

    Args:
        start (`int`): Index of the first character in the original string.
        end (`int`): Index of the character following the last character in the original string.
    """

    start: int


SUMMARY:
**Class Summary**

The `CharSpan` class represents a character span in an original string, containing the indices of its start and end points.

**Key Details**

* Primary purpose: To encapsulate the start and end indices of a substring.
* Implementation details:
	+ `start`: The index of the first character in the string.
	+ `end`: The index of the character following the last character in the string (i.e., the index after which we stop looking).
* Notable dependencies: None.

This class follows a `NamedTuple` convention, adhering to Python's PEP 489 standard for named tuple documentation.


key: chroma:document
string_value:
CODE:

class TokenSpan(NamedTuple):
    """
    Token span in an encoded string (list of tokens).

    Args:
        start (`int`): Index of the first token in the span.
        end (`int`): Index of the token following the last token in the span.
    """

    start: int


SUMMARY:
**TokenSpan Class Summary**

The `TokenSpan` class represents a span of tokens in an encoded string, with properties `start` and `end` indicating the range of indices for this span.

### Key Implementation Details:

* The class uses a `NamedTuple`, implying immutability.
* No additional methods or dependencies appear to be significant.
 
### Notable Dependencies/Requirements:

* None explicitly mentioned; however, assuming it's part of a string processing or encoding system.


key: chroma:document
string_value:
CODE:

class BatchEncoding(UserDict):
    """
    Holds the output of the [`~tokenization_utils_base.PreTrainedTokenizerBase.__call__`],
    [`~tokenization_utils_base.PreTrainedTokenizerBase.encode_plus`] and
    [`~tokenization_utils_base.PreTrainedTokenizerBase.batch_encode_plus`] methods (tokens, attention_masks, etc).

    This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes
    utility methods to map from word/character space to token space.

    Args:
        data (`dict`, *optional*):
            Dictionary of lists/arrays/tensors returned by the `__call__`/`encode_plus`/`batch_encode_plus` methods
            ('input_ids', 'attention_mask', etc.).
        encoding (`tokenizers.Encoding` or `Sequence[tokenizers.Encoding]`, *optional*):
            If the tokenizer is a fast tokenizer which outputs additional information like mapping from word/character
            space to token space the `tokenizers.Encoding` instance or list of instance (for batches) hold this
            information.
        tensor_type (`Union[None, str, TensorType]`, *optional*):
            You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at
            initialization.
        prepend_batch_axis (`bool`, *optional*, defaults to `False`):
            Whether or not to add a batch axis when converting to tensors (see `tensor_type` above). Note that this
            parameter has an effect if the parameter `tensor_type` is set, *otherwise has no effect*.
        n_sequences (`Optional[int]`, *optional*):
            You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at
            initialization.
    """

    def __init__(
        self,
        data: Optional[Dict[str, Any]] = None,
        encoding: Optional[Union[EncodingFast, Sequence[EncodingFast]]] = None,
        tensor_type: Union[None, str, TensorType] = None,
        prepend_batch_axis: bool = False,
        n_sequences: Optional[int] = None,
    ):
        super().__init__(data)

        if isinstance(encoding, EncodingFast):
            encoding = [encoding]

        self._encodings = encoding

        if n_sequences is None and encoding is not None and len(encoding):
            n_sequences = encoding[0].n_sequences

        self._n_sequences = n_sequences

        self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)

    @property
    def n_sequences(self) -> Optional[int]:
        """
        `Optional[int]`: The number of sequences used to generate each sample from the batch encoded in this
        [`BatchEncoding`]. Currently can be one of `None` (unknown), `1` (a single sentence) or `2` (a pair of
        sentences)
        """
        return self._n_sequences

    @property
    def is_fast(self) -> bool:
        """
        `bool`: Indicate whether this [`BatchEncoding`] was generated from the result of a [`PreTrainedTokenizerFast`]
        or not.
        """
        return self._encodings is not None

    def __getitem__(self, item: Union[int, str]) -> Union[Any, EncodingFast]:
        """
        If the key is a string, returns the value of the dict associated to `key` ('input_ids', 'attention_mask',
        etc.).

        If the key is an integer, get the `tokenizers.Encoding` for batch item with index `key`.

        If the key is a slice, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.)
        with the constraint of slice.
        """
        if isinstance(item, str):
            return self.data[item]
        elif self._encodings is not None:
            return self._encodings[item]
        elif isinstance(item, slice):
            return {key: self.data[key][item] for key in self.data.keys()}
        else:
            raise KeyError(
                "Invalid key. Only three types of key are available: "
                "(1) string, (2) integers for backend Encoding, and (3) slices for data subsetting."
            )

    def __getattr__(self, item: str):
        try:
            return self.data[item]
        except KeyError:
            raise AttributeError

    def __getstate__(self):
        return {"data": self.data, "encodings": self._encodings}

    def __setstate__(self, state):
        if "data" in state:
            self.data = state["data"]

        if "encodings" in state:
            self._encodings = state["encodings"]

    def keys(self):
        return self.data.keys()

    def values(self):
        return self.data.values()

    def items(self):
        return self.data.items()

    # After this point:
    # Extended properties and methods only available for fast (Rust-based) tokenizers
    # provided by HuggingFace tokenizers library.

    @property
    def encodings(self) -> Optional[List[EncodingFast]]:
        """
        `Optional[List[tokenizers.Encoding]]`: The list all encodings from the tokenization process. Returns `None` if
        the input was tokenized through Python (i.e., not a fast) tokenizer.
        """
        return self._encodings

    def tokens(self, batch_index: int = 0) -> List[str]:
        """
        Return the list of tokens (sub-parts of the input strings after word/subword splitting and before conversion to
        integer indices) at a given batch index (only works for the output of a fast tokenizer).

        Args:
            batch_index (`int`, *optional*, defaults to 0): The index to access in the batch.

        Returns:
            `List[str]`: The list of tokens at that index.
        """
        if not self._encodings:
            raise ValueError(
                "tokens() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`"
                " class)."
            )
        return self._encodings[batch_index].tokens

    def sequence_ids(self, batch_index: int = 0) -> List[Optional[int]]:
        """
        Return a list mapping the tokens to the id of their original sentences:

            - `None` for special tokens added around or between sequences,
            - `0` for tokens corresponding to words in the first sequence,
            - `1` for tokens corresponding to words in the second sequence when a pair of sequences was jointly
              encoded.

        Args:
            batch_index (`int`, *optional*, defaults to 0): The index to access in the batch.

        Returns:
            `List[Optional[int]]`: A list indicating the sequence id corresponding to each token. Special tokens added
            by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding
            sequence.
        """
        if not self._encodings:
            raise ValueError(
                "sequence_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`"
                " class)."
            )
        return self._encodings[batch_index].sequence_ids

    def words(self, batch_index: int = 0) -> List[Optional[int]]:
        """
        Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer.

        Args:
            batch_index (`int`, *optional*, defaults to 0): The index to access in the batch.

        Returns:
            `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the
            tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word
            (several tokens will be mapped to the same word index if they are parts of that word).
        """
        if not self._encodings:
            raise ValueError(
                "words() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`"
                " class)."
            )
        warnings.warn(
            "`BatchEncoding.words()` property is deprecated and should be replaced with the identical, "
            "but more self-explanatory `BatchEncoding.word_ids()` property.",
            FutureWarning,
        )
        return self.word_ids(batch_index)

    def word_ids(self, batch_index: int = 0) -> List[Optional[int]]:
        """
        Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer.

        Args:
            batch_index (`int`, *optional*, defaults to 0): The index to access in the batch.

        Returns:
            `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the
            tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word
            (several tokens will be mapped to the same word index if they are parts of that word).
        """
        if not self._encodings:
            raise ValueError(
                "word_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`"
                " class)."
            )
        return self._encodings[batch_index].word_ids

    def token_to_sequence(self, batch_or_token_index: int, token_index: Optional[int] = None) -> int:
        """
        Get the index of the sequence represented by the given token. In the general use case, this method returns `0`
        for a single sequence or the first sequence of a pair, and `1` for the second sequence of a pair

        Can be called as:

        - `self.token_to_sequence(token_index)` if batch size is 1
        - `self.token_to_sequence(batch_index, token_index)` if batch size is greater than 1

        This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e.,
        words are defined by the user). In this case it allows to easily associate encoded tokens with provided
        tokenized words.

        Args:
            batch_or_token_index (`int`):
                Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of
                the token in the sequence.
            token_index (`int`, *optional*):
                If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the
                sequence.

        Returns:
            `int`: Index of the word in the input sequence.
        """

        if not self._encodings:
            raise ValueError("token_to_sequence() is not available when using Python based tokenizers")
        if token_index is not None:
            batch_index = batch_or_token_index
        else:
            batch_index = 0
            token_index = batch_or_token_index
        if batch_index < 0:
            batch_index = self._batch_size + batch_index
        if token_index < 0:
            token_index = self._seq_len + token_index
        return self._encodings[batch_index].token_to_sequence(token_index)

    def token_to_word(self, batch_or_token_index: int, token_index: Optional[int] = None) -> int:
        """
        Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch.

        Can be called as:

        - `self.token_to_word(token_index)` if batch size is 1
        - `self.token_to_word(batch_index, token_index)` if batch size is greater than 1

        This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e.,
        words are defined by the user). In this case it allows to easily associate encoded tokens with provided
        tokenized words.

        Args:
            batch_or_token_index (`int`):
                Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of
                the token in the sequence.
            token_index (`int`, *optional*):
                If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the
                sequence.

        Returns:
            `int`: Index of the word in the input sequence.
        """

        if not self._encodings:
            raise ValueError("token_to_word() is not available when using Python based tokenizers")
        if token_index is not None:
            batch_index = batch_or_token_index
        else:
            batch_index = 0
            token_index = batch_or_token_index
        if batch_index < 0:
            batch_index = self._batch_size + batch_index
        if token_index < 0:
            token_index = self._seq_len + token_index
        return self._encodings[batch_index].token_to_word(token_index)

    def word_to_tokens(
        self, batch_or_word_index: int, word_index: Optional[int] = None, sequence_index: int = 0
    ) -> Optional[TokenSpan]:
        """
        Get the encoded token span corresponding to a word in a sequence of the batch.

        Token spans are returned as a [`~tokenization_utils_base.TokenSpan`] with:

        - **start** -- Index of the first token.
        - **end** -- Index of the token following the last token.

        Can be called as:

        - `self.word_to_tokens(word_index, sequence_index: int = 0)` if batch size is 1
        - `self.word_to_tokens(batch_index, word_index, sequence_index: int = 0)` if batch size is greater or equal to
          1

        This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words
        are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized
        words.

        Args:
            batch_or_word_index (`int`):
                Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of
                the word in the sequence.
            word_index (`int`, *optional*):
                If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the
                sequence.
            sequence_index (`int`, *optional*, defaults to 0):
                If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0
                or 1) the provided word index belongs to.

        Returns:
            ([`~tokenization_utils_base.TokenSpan`], *optional*): Span of tokens in the encoded sequence. Returns
            `None` if no tokens correspond to the word. This can happen especially when the token is a special token
            that has been used to format the tokenization. For example when we add a class token at the very beginning
            of the tokenization.
        """

        if not self._encodings:
            raise ValueError("word_to_tokens() is not available when using Python based tokenizers")
        if word_index is not None:
            batch_index = batch_or_word_index
        else:
            batch_index = 0
            word_index = batch_or_word_index
        if batch_index < 0:
            batch_index = self._batch_size + batch_index
        if word_index < 0:
            word_index = self._seq_len + word_index
        span = self._encodings[batch_index].word_to_tokens(word_index, sequence_index)
        return TokenSpan(*span) if span is not None else None

    def token_to_chars(self, batch_or_token_index: int, token_index: Optional[int] = None) -> CharSpan:
        """
        Get the character span corresponding to an encoded token in a sequence of the batch.

        Character spans are returned as a [`~tokenization_utils_base.CharSpan`] with:

        - **start** -- Index of the first character in the original string associated to the token.
        - **end** -- Index of the character following the last character in the original string associated to the
          token.

        Can be called as:

        - `self.token_to_chars(token_index)` if batch size is 1
        - `self.token_to_chars(batch_index, token_index)` if batch size is greater or equal to 1

        Args:
            batch_or_token_index (`int`):
                Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of
                the token in the sequence.
            token_index (`int`, *optional*):
                If a batch index is provided in *batch_or_token_index*, this can be the index of the token or tokens in
                the sequence.

        Returns:
            [`~tokenization_utils_base.CharSpan`]: Span of characters in the original string, or None, if the token
            (e.g. <s>, </s>) doesn't correspond to any chars in the origin string.
        """

        if not self._encodings:
            raise ValueError("token_to_chars() is not available when using Python based tokenizers")
        if token_index is not None:
            batch_index = batch_or_token_index
        else:
            batch_index = 0
            token_index = batch_or_token_index
        span_indices = self._encodings[batch_index].token_to_chars(token_index)

        return CharSpan(*span_indices) if span_indices is not None else None

    def char_to_token(
        self, batch_or_char_index: int, char_index: Optional[int] = None, sequence_index: int = 0
    ) -> int:
        """
        Get the index of the token in the encoded output comprising a character in the original string for a sequence
        of the batch.

        Can be called as:

        - `self.char_to_token(char_index)` if batch size is 1
        - `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1

        This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words
        are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized
        words.

        Args:
            batch_or_char_index (`int`):
                Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of
                the word in the sequence
            char_index (`int`, *optional*):
                If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the
                sequence.
            sequence_index (`int`, *optional*, defaults to 0):
                If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0
                or 1) the provided character index belongs to.


        Returns:
            `int`: Index of the token, or None if the char index refers to a whitespace only token and whitespace is
                   trimmed with `trim_offsets=True`.
        """

        if not self._encodings:
            raise ValueError("char_to_token() is not available when using Python based tokenizers")
        if char_index is not None:
            batch_index = batch_or_char_index
        else:
            batch_index = 0
            char_index = batch_or_char_index
        return self._encodings[batch_index].char_to_token(char_index, sequence_index)

    def word_to_chars(
        self, batch_or_word_index: int, word_index: Optional[int] = None, sequence_index: int = 0
    ) -> CharSpan:
        """
        Get the character span in the original string corresponding to given word in a sequence of the batch.

        Character spans are returned as a CharSpan NamedTuple with:

        - start: index of the first character in the original string
        - end: index of the character following the last character in the original string

        Can be called as:

        - `self.word_to_chars(word_index)` if batch size is 1
        - `self.word_to_chars(batch_index, word_index)` if batch size is greater or equal to 1

        Args:
            batch_or_word_index (`int`):
                Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of
                the word in the sequence
            word_index (`int`, *optional*):
                If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the
                sequence.
            sequence_index (`int`, *optional*, defaults to 0):
                If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0
                or 1) the provided word index belongs to.

        Returns:
            `CharSpan` or `List[CharSpan]`: Span(s) of the associated character or characters in the string. CharSpan
            are NamedTuple with:

                - start: index of the first character associated to the token in the original string
                - end: index of the character following the last character associated to the token in the original
                  string
        """

        if not self._encodings:
            raise ValueError("word_to_chars() is not available when using Python based tokenizers")
        if word_index is not None:
            batch_index = batch_or_word_index
        else:
            batch_index = 0
            word_index = batch_or_word_index
        return CharSpan(*(self._encodings[batch_index].word_to_chars(word_index, sequence_index)))

    def char_to_word(self, batch_or_char_index: int, char_index: Optional[int] = None, sequence_index: int = 0) -> int:
        """
        Get the word in the original string corresponding to a character in the original string of a sequence of the
        batch.

        Can be called as:

        - `self.char_to_word(char_index)` if batch size is 1
        - `self.char_to_word(batch_index, char_index)` if batch size is greater than 1

        This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words
        are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized
        words.

        Args:
            batch_or_char_index (`int`):
                Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of
                the character in the original string.
            char_index (`int`, *optional*):
                If a batch index is provided in *batch_or_token_index*, this can be the index of the character in the
                original string.
            sequence_index (`int`, *optional*, defaults to 0):
                If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0
                or 1) the provided character index belongs to.


        Returns:
            `int` or `List[int]`: Index or indices of the associated encoded token(s).
        """

        if not self._encodings:
            raise ValueError("char_to_word() is not available when using Python based tokenizers")
        if char_index is not None:
            batch_index = batch_or_char_index
        else:
            batch_index = 0
            char_index = batch_or_char_index
        return self._encodings[batch_index].char_to_word(char_index, sequence_index)

    def convert_to_tensors(
        self, tensor_type: Optional[Union[str, TensorType]] = None, prepend_batch_axis: bool = False
    ):
        """
        Convert the inner content to tensors.

        Args:
            tensor_type (`str` or [`~utils.TensorType`], *optional*):
                The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If
                `None`, no modification is done.
            prepend_batch_axis (`int`, *optional*, defaults to `False`):
                Whether or not to add the batch dimension during the conversion.
        """
        if tensor_type is None:
            return self

        # Convert to TensorType
        if not isinstance(tensor_type, TensorType):
            tensor_type = TensorType(tensor_type)

        # Get a function reference for the correct framework
        if tensor_type == TensorType.TENSORFLOW:
            if not is_tf_available():
                raise ImportError(
                    "Unable to convert output to TensorFlow tensors format, TensorFlow is not installed."
                )
            import tensorflow as tf

            as_tensor = tf.constant
            is_tensor = tf.is_tensor
        elif tensor_type == TensorType.PYTORCH:
            if not is_torch_available():
                raise ImportError("Unable to convert output to PyTorch tensors format, PyTorch is not installed.")
            import torch

            is_tensor = torch.is_tensor

            def as_tensor(value, dtype=None):
                if isinstance(value, list) and isinstance(value[0], np.ndarray):
                    return torch.from_numpy(np.array(value))
                return torch.tensor(value)

        elif tensor_type == TensorType.JAX:
            if not is_flax_available():
                raise ImportError("Unable to convert output to JAX tensors format, JAX is not installed.")
            import jax.numpy as jnp  # noqa: F811

            as_tensor = jnp.array
            is_tensor = is_jax_tensor

        elif tensor_type == TensorType.MLX:
            if not is_mlx_available():
                raise ImportError("Unable to convert output to MLX tensors format, MLX is not installed.")
            import mlx.core as mx

            as_tensor = mx.array

            def is_tensor(obj):
                return isinstance(obj, mx.array)
        else:

            def as_tensor(value, dtype=None):
                if isinstance(value, (list, tuple)) and isinstance(value[0], (list, tuple, np.ndarray)):
                    value_lens = [len(val) for val in value]
                    if len(set(value_lens)) > 1 and dtype is None:
                        # we have a ragged list so handle explicitly
                        value = as_tensor([np.asarray(val) for val in value], dtype=object)
                return np.asarray(value, dtype=dtype)

            is_tensor = is_numpy_array

        # Do the tensor conversion in batch
        for key, value in self.items():
            try:
                if prepend_batch_axis:
                    value = [value]

                if not is_tensor(value):
                    tensor = as_tensor(value)

                    # Removing this for now in favor of controlling the shape with `prepend_batch_axis`
                    # # at-least2d
                    # if tensor.ndim > 2:
                    #     tensor = tensor.squeeze(0)
                    # elif tensor.ndim < 2:
                    #     tensor = tensor[None, :]

                    self[key] = tensor
            except Exception as e:
                if key == "overflowing_tokens":
                    raise ValueError(
                        "Unable to create tensor returning overflowing tokens of different lengths. "
                        "Please see if a fast version of this tokenizer is available to have this feature available."
                    ) from e
                raise ValueError(
                    "Unable to create tensor, you should probably activate truncation and/or padding with"
                    " 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your"
                    f" features (`{key}` in this case) have excessive nesting (inputs type `list` where type `int` is"
                    " expected)."
                ) from e

        return self

    def to(self, device: Union[str, "torch.device"], *, non_blocking: bool = False) -> "BatchEncoding":
        """
        Send all values to device by calling `v.to(device, non_blocking=non_blocking)` (PyTorch only).

        Args:
            device (`str` or `torch.device`): The device to put the tensors on.
            non_blocking (`bool`): Whether to perform the copy asynchronously.

        Returns:
            [`BatchEncoding`]: The same instance after modification.
        """
        requires_backends(self, ["torch"])
        import torch

        # This check catches things like APEX blindly calling "to" on all inputs to a module
        # Otherwise it passes the casts down and casts the LongTensor containing the token idxs
        # into a HalfTensor
        if isinstance(device, str) or is_torch_device(device) or isinstance(device, int):
            self.data = {
                k: v.to(device=device, non_blocking=non_blocking) if isinstance(v, torch.Tensor) else v
                for k, v in self.data.items()
            }
        else:
            logger.warning(f"Attempting to cast a BatchEncoding to type {str(device)}. This is not supported.")


SUMMARY:
This is a Python code snippet that defines a class `BatchEncoding` used for encoding batch data in a tokenizer. The class has methods for converting the output to different tensor formats, including TensorFlow, PyTorch, and JAX, as well as methods for adding and extracting values, accessing and modifying token lengths, handling overflowing tokens, truncation, padding, and device transfer.

Key Features:

1. **TensorConversion**: The `to` method converts the batch encoding to a specified device (e.g., GPU or CPU) using a given framework.
2. **Value extraction**: The `items`, `__getitem__`, and `get_values` methods allow extracting specific values from the batch encoding.
3. **Token Lengths**: The `tokens` property provides access to the token lengths in the batch, and the `set_token_lengths` method updates these lengths.
4. **Overflowing Tokens Handling**: The `overflowing_tokens` property tracks tokens that exceed a specified length limit using bitwise flags.
5. **Truncation and Padding**: The `truncation` and `padding` properties control whether truncation of overflowing tokens is enabled, and padding is applied to the tokenizer's output.

Method Signatures:

- `get_values`
- `set_token_lengths`
- `overflowing_tokens=`
- `__getitem__
- `items`
- `to`

Overall, this class provides a comprehensive structure for batch encoding, value extraction, and tensor conversion across different frameworks.


key: chroma:document
string_value:
CODE:

class SpecialTokensMixin:
    """
    A mixin derived by [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`] to handle specific behaviors related to
    special tokens. In particular, this class hold the attributes which can be used to directly access these special
    tokens in a model-independent manner and allow to set and update the special tokens.

    Args:
        bos_token (`str` or `tokenizers.AddedToken`, *optional*):
            A special token representing the beginning of a sentence.
        eos_token (`str` or `tokenizers.AddedToken`, *optional*):
            A special token representing the end of a sentence.
        unk_token (`str` or `tokenizers.AddedToken`, *optional*):
            A special token representing an out-of-vocabulary token.
        sep_token (`str` or `tokenizers.AddedToken`, *optional*):
            A special token separating two different sentences in the same input (used by BERT for instance).
        pad_token (`str` or `tokenizers.AddedToken`, *optional*):
            A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by
            attention mechanisms or loss computation.
        cls_token (`str` or `tokenizers.AddedToken`, *optional*):
            A special token representing the class of the input (used by BERT for instance).
        mask_token (`str` or `tokenizers.AddedToken`, *optional*):
            A special token representing a masked token (used by masked-language modeling pretraining objectives, like
            BERT).
        additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*):
            A tuple or a list of additional tokens, which will be marked as `special`, meaning that they will be
            skipped when decoding if `skip_special_tokens` is set to `True`.
    """

    SPECIAL_TOKENS_ATTRIBUTES = [
        "bos_token",
        "eos_token",
        "unk_token",
        "sep_token",
        "pad_token",
        "cls_token",
        "mask_token",
        "additional_special_tokens",
    ]

    def __init__(self, verbose=False, **kwargs):
        self._pad_token_type_id = 0
        self.verbose = verbose
        self._special_tokens_map = {attr: None for attr in self.SPECIAL_TOKENS_ATTRIBUTES}
        self._special_tokens_map["additional_special_tokens"] = []  # for BC where it defaults to empty list

        # We directly set the hidden value to allow initialization with special tokens
        # which are not yet in the vocabulary. Necessary for serialization/de-serialization
        # TODO clean this up at some point (probably by switching to fast tokenizers)

        for key, value in kwargs.items():
            if value is None:
                continue
            if key in self.SPECIAL_TOKENS_ATTRIBUTES:
                if key == "additional_special_tokens":
                    assert isinstance(value, (list, tuple)), f"Value {value} is not a list or tuple"
                    assert all(
                        isinstance(t, (str, AddedToken)) for t in value
                    ), "One of the tokens is not a string or an AddedToken"
                    setattr(self, key, value)
                elif isinstance(value, (str, AddedToken)):
                    setattr(self, key, value)
                else:
                    raise TypeError(f"Special token {key} has to be either str or AddedToken but got: {type(value)}")

    def sanitize_special_tokens(self) -> int:
        """
        The `sanitize_special_tokens` is now deprecated kept for backward compatibility and will be removed in
        transformers v5.
        """
        logger.warning_once("The `sanitize_special_tokens` will be removed in transformers v5.")
        return self.add_tokens(self.all_special_tokens_extended, special_tokens=True)

    def add_special_tokens(
        self, special_tokens_dict: Dict[str, Union[str, AddedToken]], replace_additional_special_tokens=True
    ) -> int:
        """
        Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If
        special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the
        current vocabulary).

        When adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the
        model so that its embedding matrix matches the tokenizer.

        In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method.

        Using `add_special_tokens` will ensure your special tokens can be used in several ways:

        - Special tokens can be skipped when decoding using `skip_special_tokens = True`.
        - Special tokens are carefully handled by the tokenizer (they are never split), similar to `AddedTokens`.
        - You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This
          makes it easy to develop model-agnostic training and fine-tuning scripts.

        When possible, special tokens are already registered for provided pretrained models (for instance
        [`BertTokenizer`] `cls_token` is already registered to be :obj*'[CLS]'* and XLM's one is also registered to be
        `'</s>'`).

        Args:
            special_tokens_dict (dictionary *str* to *str* or `tokenizers.AddedToken`):
                Keys should be in the list of predefined special attributes: [`bos_token`, `eos_token`, `unk_token`,
                `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`].

                Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer
                assign the index of the `unk_token` to them).
            replace_additional_special_tokens (`bool`, *optional*,, defaults to `True`):
                If `True`, the existing list of additional special tokens will be replaced by the list provided in
                `special_tokens_dict`. Otherwise, `self._special_tokens_map["additional_special_tokens"]` is just extended. In the former
                case, the tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged
                as non-special tokens. Remember, this only affects which tokens are skipped during decoding, not the
                `added_tokens_encoder` and `added_tokens_decoder`. This means that the previous
                `additional_special_tokens` are still added tokens, and will not be split by the model.

        Returns:
            `int`: Number of tokens added to the vocabulary.

        Examples:

        ```python
        # Let's see how to add a new classification token to GPT-2
        tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2")
        model = GPT2Model.from_pretrained("openai-community/gpt2")

        special_tokens_dict = {"cls_token": "<CLS>"}

        num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
        print("We have added", num_added_toks, "tokens")
        # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.
        model.resize_token_embeddings(len(tokenizer))

        assert tokenizer.cls_token == "<CLS>"
        ```"""
        if not special_tokens_dict:
            return 0

        added_tokens = []
        for key, value in special_tokens_dict.items():
            assert key in self.SPECIAL_TOKENS_ATTRIBUTES, f"Key {key} is not a special token"

            if self.verbose:
                logger.info(f"Assigning {value} to the {key} key of the tokenizer")

            if key == "additional_special_tokens":
                assert isinstance(value, (list, tuple)) and all(
                    isinstance(t, (str, AddedToken)) for t in value
                ), f"Tokens {value} for key {key} should all be str or AddedToken instances"

                to_add = []
                for token in value:
                    if isinstance(token, str):
                        # for legacy purpose we default to stripping. `test_add_tokens_tokenizer` depends on this
                        token = AddedToken(token, rstrip=False, lstrip=False, normalized=False, special=True)
                    if not replace_additional_special_tokens and str(token) in self.additional_special_tokens:
                        continue
                    to_add.append(token)
                if replace_additional_special_tokens and len(to_add) > 0:
                    setattr(self, key, list(to_add))
                else:
                    self._special_tokens_map["additional_special_tokens"].extend(to_add)
                added_tokens += to_add

            else:
                if not isinstance(value, (str, AddedToken)):
                    raise ValueError(f"Token {value} for key {key} should be a str or an AddedToken instance")
                if isinstance(value, (str)):
                    # for legacy purpose we default to stripping. `False` depends on this
                    value = AddedToken(value, rstrip=False, lstrip=False, normalized=False, special=True)
                if isinstance(value, AddedToken):
                    setattr(self, key, value)
                if value not in added_tokens:
                    added_tokens.append(value)

        # if we are adding tokens that were not part of the vocab, we ought to add them
        added_tokens = self.add_tokens(added_tokens, special_tokens=True)
        return added_tokens

    def add_tokens(
        self, new_tokens: Union[str, AddedToken, List[Union[str, AddedToken]]], special_tokens: bool = False
    ) -> int:
        """
        Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to
        it with indices starting from length of the current vocabulary and will be isolated before the tokenization
        algorithm is applied. Added tokens and tokens from the vocabulary of the tokenization algorithm are therefore
        not treated in the same way.

        Note, when adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix
        of the model so that its embedding matrix matches the tokenizer.

        In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method.

        Args:
            new_tokens (`str`, `tokenizers.AddedToken` or a list of *str* or `tokenizers.AddedToken`):
                Tokens are only added if they are not already in the vocabulary. `tokenizers.AddedToken` wraps a string
                token to let you personalize its behavior: whether this token should only match against a single word,
                whether this token should strip all potential whitespaces on the left side, whether this token should
                strip all potential whitespaces on the right side, etc.
            special_tokens (`bool`, *optional*, defaults to `False`):
                Can be used to specify if the token is a special token. This mostly change the normalization behavior
                (special tokens like CLS or [MASK] are usually not lower-cased for instance).

                See details for `tokenizers.AddedToken` in HuggingFace tokenizers library.

        Returns:
            `int`: Number of tokens added to the vocabulary.

        Examples:

        ```python
        # Let's see how to increase the vocabulary of Bert model and tokenizer
        tokenizer = BertTokenizerFast.from_pretrained("google-bert/bert-base-uncased")
        model = BertModel.from_pretrained("google-bert/bert-base-uncased")

        num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"])
        print("We have added", num_added_toks, "tokens")
        # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.
        model.resize_token_embeddings(len(tokenizer))
        ```"""
        if not new_tokens:
            return 0

        if not isinstance(new_tokens, (list, tuple)):
            new_tokens = [new_tokens]

        return self._add_tokens(new_tokens, special_tokens=special_tokens)

    def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int:
        raise NotImplementedError

    @property
    def pad_token_type_id(self) -> int:
        """
        `int`: Id of the padding token type in the vocabulary.
        """
        return self._pad_token_type_id

    def __setattr__(self, key, value):
        key_without_id = key
        key_is_special_id = key.endswith("_id") or key.endswith("_ids")
        if key_is_special_id:
            key_without_id = key[:-3] if not key.endswith("_ids") else key[:-4]

        if self.__dict__.get("_special_tokens_map", None) is not None and any(
            name in self.__dict__["_special_tokens_map"] for name in [key, key_without_id]
        ):
            if key_is_special_id:
                if value is not None:
                    value = (
                        self.convert_ids_to_tokens(value)
                        if key != "additional_special_tokens"
                        else [self.convert_ids_to_tokens(val) for val in value]
                    )
                key = key_without_id

            if key != "additional_special_tokens" and not isinstance(value, (str, AddedToken)) and value is not None:
                raise ValueError(f"Cannot set a non-string value as the {key}")
            self._special_tokens_map[key] = value
        else:
            super().__setattr__(key, value)

    def __getattr__(self, key):
        key_without_id = key
        key_is_special_id = key.endswith("_id") or key.endswith("_ids")
        if key_is_special_id:
            key_without_id = key[:-3] if not key.endswith("_ids") else key[:-4]

        if self.__dict__.get("_special_tokens_map", None) is not None and any(
            name in self.__dict__["_special_tokens_map"] for name in [key, key_without_id]
        ):
            _special_tokens_map = self.__dict__["_special_tokens_map"]
            if not key_is_special_id:
                if _special_tokens_map[key] is None:
                    if self.verbose:
                        logger.error(f"Using {key}, but it is not set yet.")
                    return None
                value = _special_tokens_map[key]
                return str(value) if key != "additional_special_tokens" else [str(tok) for tok in value]
            else:
                attr_as_tokens = getattr(self, key_without_id)
                return self.convert_tokens_to_ids(attr_as_tokens) if attr_as_tokens is not None else None

        if key not in self.__dict__:
            raise AttributeError(f"{self.__class__.__name__} has no attribute {key}")
        else:
            return super().__getattr__(key)

    @property
    def special_tokens_map(self) -> Dict[str, Union[str, List[str]]]:
        """
        `Dict[str, Union[str, List[str]]]`: A dictionary mapping special token class attributes (`cls_token`,
        `unk_token`, etc.) to their values (`'<unk>'`, `'<cls>'`, etc.).

        Convert potential tokens of `tokenizers.AddedToken` type to string.
        """
        set_attr = {}
        for attr in self.SPECIAL_TOKENS_ATTRIBUTES:
            attr_value = getattr(self, attr)
            if attr_value:
                set_attr[attr] = attr_value
        return set_attr

    @property
    def special_tokens_map_extended(self) -> Dict[str, Union[str, AddedToken, List[Union[str, AddedToken]]]]:
        """
        `Dict[str, Union[str, tokenizers.AddedToken, List[Union[str, tokenizers.AddedToken]]]]`: A dictionary mapping
        special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`'<unk>'`, `'<cls>'`, etc.).

        Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how
        special tokens are tokenized.
        """
        set_attr = {}
        for attr in self.SPECIAL_TOKENS_ATTRIBUTES:
            attr_value = self._special_tokens_map[attr]
            if attr_value:
                set_attr[attr] = attr_value
        return set_attr

    @property
    def all_special_tokens_extended(self) -> List[Union[str, AddedToken]]:
        """
        `List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`'<unk>'`, `'<cls>'`, etc.), the order has
        nothing to do with the index of each tokens. If you want to know the correct indices, check
        `self.added_tokens_encoder`. We can't create an order anymore as the keys are `AddedTokens` and not `Strings`.

        Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how
        special tokens are tokenized.
        """
        all_tokens = []
        seen = set()
        for value in self.special_tokens_map_extended.values():
            if isinstance(value, (list, tuple)):
                tokens_to_add = [token for token in value if str(token) not in seen]
            else:
                tokens_to_add = [value] if str(value) not in seen else []
            seen.update(map(str, tokens_to_add))
            all_tokens.extend(tokens_to_add)
        return all_tokens

    @property
    def all_special_tokens(self) -> List[str]:
        """
        `List[str]`: A list of the unique special tokens (`'<unk>'`, `'<cls>'`, ..., etc.).

        Convert tokens of `tokenizers.AddedToken` type to string.
        """
        all_toks = [str(s) for s in self.all_special_tokens_extended]
        return all_toks

    @property
    def all_special_ids(self) -> List[int]:
        """
        `List[int]`: List the ids of the special tokens(`'<unk>'`, `'<cls>'`, etc.) mapped to class attributes.
        """
        all_toks = self.all_special_tokens
        all_ids = self.convert_tokens_to_ids(all_toks)
        return all_ids

    def _set_model_specific_special_tokens(self, special_tokens: List[str]):
        """
        Adds new special tokens to the "SPECIAL_TOKENS_ATTRIBUTES" list which will be part
        of "self.special_tokens" and saved as a special token in tokenizer's config.
        This allows us to dynamically add new model-type specific tokens after initilizing the tokenizer.
        For example: if the model tokenizers is multimodal, we can support special image or audio tokens.
        """
        self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
        for key, value in special_tokens.items():
            if isinstance(value, (str, AddedToken)):
                self._special_tokens_map[key] = value
            else:


SUMMARY:
**Summary**

This class represents a tokenizer, which is a text preprocessing technique used in natural language processing (NLP) tasks. The tokenizer breaks down input text into subwords (smaller units of words) and assigns unique identifiers (ids) to each token.

Key features:

*   Tokenization: Breaks down input text into individual tokens.
*   Subwording: Uses a vocabulary list to identify the most representative word for a given sequence of characters.
*   Special tokens: Supports special tokens like [CLS], [UNK], and [SEP] with their corresponding IDs.
*   Model-specific special tokens: Allows models to dynamically add new model-type specific tokens.

Methods:

*   `__init__(self, ...):`: Initializes the tokenizer with various parameters like vocab_size, max_token_length, etc.
*   `encode(self, text, return_attention_mask=False, return_tensors=None):`: Encodes inputs into a format suitable for the input layer of a transformer model. If no special tokens were seen when decoding input, it returns `[CLS]`.
*   `get_vocab(self)`: Returns the vocabulary with ids starting from 0 (not -1).
*   `is_vocab_token(token, vocab, include SpecialTokens=True):`: Checks whether a given token in different environments is a valid token.
*   `convert_tokens_to_ids(tokens, skip_special_tokens=False,...):`: Converts input tokens to the corresponding IDs in this model tokenizer representation.

Example use cases:
```python
# Create a tokenizer from Hugging Face's Transformers library
from transformers import Tokenizer

tokenizer = Tokenizer.from_pretrained('bert-base-uncased', special_tokens=['[CLS]'])

# Encode input text into token IDs and attention masks
input_text = 'This is an example sentence.'
token_ids, attention_mask = tokenizer.encode(input_text, return AttentionMask=True)

# Get the vocabulary with IDs starting from 0 (not -1)
vocab = tokenizer.get_vocab()

print(vocab)

# Convert tokens to IDs
token_ids = tokenizer.convert_tokens_to_ids(tokenizer.encode('Hello World'))
print(token_ids)
```
The provided class provides extensive functionality for processing and tokenizing text data in a robust manner. The detailed documentation showcases its capabilities, allowing developers to effectively use the class for their specific NLP requirements.


key: chroma:document
string_value:
CODE:

def get_fast_tokenizer_file(tokenization_files: List[str]) -> str:
    """
    Get the tokenization file to use for this version of transformers.

    Args:
        tokenization_files (`List[str]`): The list of available configuration files.

    Returns:
        `str`: The tokenization file to use.
    """
    tokenizer_files_map = {}
    for file_name in tokenization_files:
        search = _re_tokenizer_file.search(file_name)
        if search is not None:
            v = search.groups()[0]
            tokenizer_files_map[v] = file_name
    available_versions = sorted(tokenizer_files_map.keys())

    # Defaults to FULL_TOKENIZER_FILE and then try to look at some newer versions.
    tokenizer_file = FULL_TOKENIZER_FILE
    transformers_version = version.parse(__version__)
    for v in available_versions:
        if version.parse(v) <= transformers_version:
            tokenizer_file = tokenizer_files_map[v]
        else:
            # No point going further since the versions are sorted.
            break



SUMMARY:
**Function Summary: `get_fast_tokenizer_file`**

**Purpose:** Returns the tokenization file to use for the current version of transformers based on available configuration files.

**Implementation Details:**

* Iterates through a list of tokenization files (`tokenization_files`) and extracts model versions from each file name using `_re_tokenizer_file.search(file_name)`.
* Stores the mapping between model versions and corresponding file names in `tokenizer_files_map`.
* Sorts the available model versions and defaults to an older version (`FULL_TOKENIZER_FILE`) unless a newer version is found.

**Key Method Signature:**
`get_fast_tokenizer_file(tokenization_files: List[str]) -> str`

**Notable Dependencies:**

* `_re_tokenizer_file`: Regular expression used to extract model versions from file names.
* `version.parse`: Used to parse and compare model version strings.


key: chroma:document
string_value:
CODE:

def dummy_loss(y_true, y_pred):
    if y_pred.shape.rank <= 1:
        return y_pred
    else:
        reduction_axes = list(range(1, y_pred.shape.rank))


SUMMARY:
**Summary:**

This function calculates the mean of predicted probabilities (`y_pred`) across multiple axes if they have more than one dimension. If `y_pred` has only one dimension (singleton value), it returns the raw value.

**Key Implementation Details:**

* Calculates mean reduction using TensorFlow (`tf.reduce_mean`)
* Reduces across all axes except the batch axis (index 0)
* Returns a scalar value

**Notable Dependencies and Requirements:**

* TensorFlow library (`tf`)


key: chroma:document
string_value:
CODE:

class TFModelUtilsMixin:
    """
    A few utilities for `keras.Model`, to be used as a mixin.
    """

    def num_parameters(self, only_trainable: bool = False) -> int:
        """
        Get the number of (optionally, trainable) parameters in the model.

        Args:
            only_trainable (`bool`, *optional*, defaults to `False`):
                Whether or not to return only the number of trainable parameters

        Returns:
            `int`: The number of parameters.
        """
        if only_trainable:
            return int(sum(np.prod(w.shape.as_list()) for w in self.trainable_variables))
        else:


SUMMARY:
**TFModelUtilsMixin Class Summary**

### Purpose and Functionality

This mixin provides utility methods for `keras.Model` instances. Specifically, it offers a method to estimate the number of model parameters.

### Key Implementation Details

*   The `num_parameters` method returns the total or trainable number of model parameters based on the `only_trainable` flag.
*   The method uses the `count_params()` and `trainable_variables` attributes of the `keras.Model`.

### Notable Dependencies and Requirements

*   Uses `numpy` for vectorized operations (`np.prod`, `np.shape.as_list()`)
*   Inherently expects a `keras.Model` instance with a defined `trainable_variables` attribute.


key: chroma:document
string_value:
CODE:

def keras_serializable(cls):
    """
    Decorate a Keras Layer class to support Keras serialization.

    This is done by:

    1. Adding a `transformers_config` dict to the Keras config dictionary in `get_config` (called by Keras at
       serialization time.
    2. Wrapping `__init__` to accept that `transformers_config` dict (passed by Keras at deserialization time) and
       convert it to a config object for the actual layer initializer.
    3. Registering the class as a custom object in Keras (if the Tensorflow version supports this), so that it does not
       need to be supplied in `custom_objects` in the call to `keras.models.load_model`.

    Args:
        cls (a `keras.layers.Layers subclass`):
            Typically a `TF.MainLayer` class in this project, in general must accept a `config` argument to its
            initializer.

    Returns:
        The same class object, with modifications for Keras deserialization.
    """
    initializer = cls.__init__

    config_class = getattr(cls, "config_class", None)
    if config_class is None:
        raise AttributeError("Must set `config_class` to use @keras_serializable")

    @functools.wraps(initializer)
    def wrapped_init(self, *args, **kwargs):
        config = args[0] if args and isinstance(args[0], PretrainedConfig) else kwargs.pop("config", None)

        if isinstance(config, dict):
            config = config_class.from_dict(config)
            initializer(self, config, *args, **kwargs)
        elif isinstance(config, PretrainedConfig):
            if len(args) > 0:
                initializer(self, *args, **kwargs)
            else:
                initializer(self, config, *args, **kwargs)
        else:
            raise ValueError("Must pass either `config` (PretrainedConfig) or `config` (dict)")

        self._config = config
        self._kwargs = kwargs

    cls.__init__ = wrapped_init

    if not hasattr(cls, "get_config"):
        raise TypeError("Only use @keras_serializable on keras.layers.Layer subclasses")
    if hasattr(cls.get_config, "_is_default"):

        def get_config(self):
            cfg = super(cls, self).get_config()
            cfg["config"] = self._config.to_dict()
            cfg.update(self._kwargs)
            return cfg

        cls.get_config = get_config

    cls._keras_serializable = True
    if hasattr(keras.utils, "register_keras_serializable"):
        cls = keras.utils.register_keras_serializable()(cls)


SUMMARY:
**Functionality**

This function `keras_serializable` decorates a Keras Layer class to support Keras serialization. It:

1. Adds a transformation config dictionary to the Keras config dictionary atserialization time.
2. Wraps `__init__` to accept and convert the transformation config to a config object for deserialization.
3. Registers the class as a custom object in Keras.

**Key Implementation Details**

*   Wrapping `__init__` to handle serialization-related args and conversions.
*   Adding `transformers_config` dict to Keras config dictionary at serialization time.
*   Updating instance attributes with converted `_config` object and passed `kwargs`.

**Notable Dependencies or Requirements**

*   Keras (`layers.Layers subclass`)
*   TensorFlow (`keras.layers.MainLayer`, `PretrainedConfig`)


key: chroma:document
string_value:
CODE:

class TFCausalLanguageModelingLoss:
    """
    Loss function suitable for causal language modeling (CLM), that is, the task of guessing the next token.

    <Tip>

    Any label of -100 will be ignored (along with the corresponding logits) in the loss computation.

    </Tip>
    """

    def hf_compute_loss(self, labels, logits):
        loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction=keras.losses.Reduction.NONE)
        if self.config.tf_legacy_loss:
            # make sure only labels that are not equal to -100 affect the loss
            active_loss = tf.not_equal(tf.reshape(labels, (-1,)), -100)
            reduced_logits = tf.boolean_mask(tf.reshape(logits, (-1, shape_list(logits)[2])), active_loss)
            labels = tf.boolean_mask(tf.reshape(labels, (-1,)), active_loss)
            return loss_fn(labels, reduced_logits)

        # Clip negative labels to zero here to avoid NaNs and errors - those positions will get masked later anyway
        unmasked_loss = loss_fn(tf.nn.relu(labels), logits)
        # make sure only labels that are not equal to -100 affect the loss
        loss_mask = tf.cast(labels != -100, dtype=unmasked_loss.dtype)
        masked_loss = unmasked_loss * loss_mask
        reduced_masked_loss = tf.reduce_sum(masked_loss) / tf.reduce_sum(loss_mask)


SUMMARY:
**Class Summary: TFCausalLanguageModelingLoss**

**Purpose and Functionality:**
TFCausalLanguageModelingLoss is a loss function designed for causal language modeling tasks. It measures the difference between predicted logits and true labels.

**Key Implementation Details:**

* Applies sparse categorical cross-entropy loss with `from_logits=True` and `reduction=NONE`.
* Ignores labels of -100 in loss computation by applying masking.
* Avoids NaNs by clipping negative labels to zero.

**Method Signatures:**

* `hf_compute_loss(labels, logits)`: computes the loss using either TF legacy methods or optimized unmasked approach.

**Notable Dependencies:**

* Keras (`keras.losses.SparseCategoricalCrossentropy`)
* TensorFlow (`tf`, `tf.not_equal`, `tf.boolean_mask`)
* Utility functions (e.g., `shape_list`)

Note: The class has a configuration option (`self.config.tf_legacy_loss`) that determines whether to use the TF legacy loss computation approach or the optimized unmasked approach.


key: chroma:document
string_value:
CODE:

class TFQuestionAnsweringLoss:
    """
    Loss function suitable for question answering.
    """

    def hf_compute_loss(self, labels, logits):
        loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction=keras.losses.Reduction.NONE)
        start_loss = loss_fn(labels["start_position"], logits[0])
        end_loss = loss_fn(labels["end_position"], logits[1])



SUMMARY:
**TFQuestionAnsweringLoss Class Summary**

### Purpose and Functionality

This class implements the TFQuestionAnsweringLoss, a loss function suitable for question answering tasks.

It takes in `labels` and `logits` as inputs and computes the average of two components:

1.  `start_loss`: computed using `SparseCategoricalCrossentropy` with the "start_position" class label and the first element of the logits tensor.
2.  `end_loss`: computed similarly but for the "end_position" class label and the second element of the logits tensor.

### Key Implementation Details

*   Uses the `SparseCategoricalCrossentropy` loss function from TensorFlow (`keras.losses`) with `from_logits=True` and `reduction=none`.
*   The average of `start_loss` and `end_loss` is returned to compute the total loss.

### Notable Dependencies or Requirements

*   External Dependency: `keras` (specifically, `keras.losses.SparseCategoricalCrossentropy`) 

### Method Signatures and Interfaces

*   `hf_compute_loss(labels, logits)`: a method that takes in `labels` dictionary and `logits` tensor as input and returns the computed loss value.

 Note that no interfaces are implemented by this class.


key: chroma:document
string_value:
CODE:

class TFTokenClassificationLoss:
    """
    Loss function suitable for token classification.

    <Tip>

    Any label of -100 will be ignored (along with the corresponding logits) in the loss computation.

    </Tip>
    """

    def hf_compute_loss(self, labels, logits):
        loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction=keras.losses.Reduction.NONE)
        if tf.executing_eagerly():  # Data-dependent conditionals are forbidden in XLA
            if tf.math.reduce_any(labels == -1):
                tf.print("Using `-1` to mask the loss for the token is deprecated. Please use `-100` instead.")

        if self.config.tf_legacy_loss:
            # make sure only labels that are not equal to -100
            # are taken into account as loss
            if tf.math.reduce_any(labels == -1):
                tf.print("Using `-1` to mask the loss for the token is deprecated. Please use `-100` instead.")
                active_loss = tf.reshape(labels, (-1,)) != -1
            else:
                active_loss = tf.reshape(labels, (-1,)) != -100
            reduced_logits = tf.boolean_mask(tf.reshape(logits, (-1, shape_list(logits)[2])), active_loss)
            labels = tf.boolean_mask(tf.reshape(labels, (-1,)), active_loss)

            return loss_fn(labels, reduced_logits)

        # Clip negative labels to zero here to avoid NaNs and errors - those positions will get masked later anyway
        unmasked_loss = loss_fn(tf.nn.relu(labels), logits)
        # make sure only labels that are not equal to -100 or -1
        # are taken into account as loss
        loss_mask = tf.cast(labels >= 0, dtype=unmasked_loss.dtype)
        # Avoid possible division by zero later
        # Masked positions will have a loss of NaN because -100 and -1 are not valid labels
        masked_loss = unmasked_loss * loss_mask
        reduced_masked_loss = tf.reduce_sum(masked_loss) / tf.reduce_sum(loss_mask)


SUMMARY:
**TFTokenClassificationLoss Class Summary**

### Purpose and Functionality

`TFTokenClassificationLoss` is a loss function designed for token classification tasks in NLP. It computes the sparse categorical cross-entropy loss while ignoring labels of -100.

### Key Implementation Details

*   Uses `keras.losses.SparseCategoricalCrossentropy` with `from_logits=True` and `reduction=keras.losses.Reduction.NONE`.
*   Checks for data-dependent conditions like using `-1` to mask the loss, which is deprecated; recommending instead using `-100`.

### Notable Dependencies or Requirements

*   **keras** library
*   **tf** library (with XLA support)
*   `sparse_categorical_crossentropy` function from `keras.losses`
*   `math` and `print` functions from `tf`


key: chroma:document
string_value:
CODE:

class TFSequenceClassificationLoss:
    """
    Loss function suitable for sequence classification.
    """

    def hf_compute_loss(self, labels, logits):
        if logits.shape.rank == 1 or logits.shape[1] == 1:
            loss_fn = keras.losses.MeanSquaredError(reduction=keras.losses.Reduction.NONE)
            if labels.shape.rank == 1:
                # MeanSquaredError returns a scalar loss if the labels are 1D, so avoid that
                labels = tf.expand_dims(labels, axis=-1)
        else:
            loss_fn = keras.losses.SparseCategoricalCrossentropy(
                from_logits=True, reduction=keras.losses.Reduction.NONE
            )



SUMMARY:
**Summary:**

The `TFSequenceClassificationLoss` class computes the sequence classification loss using mean squared error (MSE) or sparse categorical cross-entropy (SCCE). 

* **Primary Purpose:** Computes sequence classification loss based on input labels and logits.

**Key Implementation Details:**

* Uses MSE with `Reduction.NONE` if `labels` are 1D.
* Otherwise, uses SCCE with `from_logits=True` and `Reduction.NONE`.

**Notable Dependencies:**

* Keras (`mean_squared_error`, `sparse_categorical_crossentropy`)
* TensorFlow (`tf`)


key: chroma:document
string_value:
CODE:

class TFMultipleChoiceLoss:
    """Loss function suitable for multiple choice tasks."""

    def hf_compute_loss(self, labels, logits):
        loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction=keras.losses.Reduction.NONE)


SUMMARY:
**TFMultipleChoiceLoss Class Summary**

### Main Purpose and Functionality
The `TFMultipleChoiceLoss` class implements a loss function suitable for multiple choice tasks using the sparse categorical cross-entropy (SCCE) metric from TensorFlow's Keras API.

### Key Implementation Details

* Uses `keras.losses.SparseCategoricalCrossentropy` with `from_logits=True` and `reduction=keras.losses.Reduction.NONE`.
* Computed loss is returned after applying the SCCE metric to labeled outputs (`labels`) and logits (`logits`). 

### Notable Dependencies or Requirements
- `keras` (TensorFlow's Keras API)
- `keras.losses.SparseCategoricalCrossentropy`

### Method Signatures and Interfaces

* `hf_compute_loss(labels, logits)`: Computes the loss using SCCE metric.

### Inherited or Implemented Interfaces
None


key: chroma:document
string_value:
CODE:

class TFMaskedLanguageModelingLoss(TFCausalLanguageModelingLoss):
    """
    Loss function suitable for masked language modeling (MLM), that is, the task of guessing the masked tokens.

    <Tip>

    Any label of -100 will be ignored (along with the corresponding logits) in the loss computation.

    </Tip>


SUMMARY:
**Summary**

TFMaskedLanguageModelingLoss: A custom loss function for masked language modeling, inheriting from TFCausalLanguageModelingLoss. Ignores labels and corresponding logits with value -100 in the loss computation. 

**Key Implementation Details:**

* Inherits from `TFCausalLanguageModelingLoss`
* Ignores labels with `value -100` in the loss computation

**Notable Dependencies:**

* `TFCausalLanguageModelingLoss` (base class)
* None explicitly listed


key: chroma:document
string_value:
CODE:

class TFNextSentencePredictionLoss:
    """
    Loss function suitable for next sentence prediction (NSP), that is, the task of guessing the next sentence.

    <Tip>

    Any label of -100 will be ignored (along with the corresponding logits) in the loss computation.

    </Tip>
    """

    def hf_compute_loss(self, labels, logits):
        loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction=keras.losses.Reduction.NONE)
        if self.config.tf_legacy_loss:
            # make sure only labels that are not equal to -100
            # are taken into account as loss
            next_sentence_active_loss = tf.not_equal(tf.reshape(labels, (-1,)), -100)
            next_sentence_reduced_logits = tf.boolean_mask(tf.reshape(logits, (-1, 2)), next_sentence_active_loss)
            next_sentence_label = tf.boolean_mask(tf.reshape(labels, (-1,)), next_sentence_active_loss)

            return loss_fn(next_sentence_label, next_sentence_reduced_logits)

        # make sure only labels that are not equal to -100
        # are taken into account as loss

        # Clip negative labels to zero here to avoid NaNs and errors - those positions will get masked later anyway
        unmasked_ns_loss = loss_fn(y_true=tf.nn.relu(labels), y_pred=logits)
        ns_loss_mask = tf.cast(labels != -100, dtype=unmasked_ns_loss.dtype)
        # Just zero out samples where label is -100, no reduction
        masked_ns_loss = unmasked_ns_loss * ns_loss_mask



SUMMARY:
**Summary:**

`TFNextSentencePredictionLoss` class calculates the next sentence prediction loss. This loss function is suitable for next sentence prediction (NSP) tasks.

**Main Purpose and Functionality:**

* Calculates the loss between predicted logits and true labels.
* Ignores labels of -100 along with corresponding logits.

**Key Implementation Details:**

* Uses `SparseCategoricalCrossentropy` loss function.
* Applies mask to ignore invalid labels.
* Returns masked loss values.

**Important Method Signatures and Interfaces:**

* `hf_compute_loss(labels, logits)`: main calculation method for next sentence prediction loss.

**Notable Dependencies and Requirements:**

* Keras (`losses.SparseCategoricalCrossentropy`)
* TensorFlow (TF)
* TensorFlow 1.x compatibility (through `tf` namespace)

**Additional Notes:**

* Label values of -100 are ignored in the loss computation.
* Loss calculation uses a combination of masking and reduction to handle invalid labels.


key: chroma:document
string_value:
CODE:

def booleans_processing(config, **kwargs):
    """
    Process the input booleans of each model.

    Args:
        config ([`PretrainedConfig`]):
            The config of the running model.
        **kwargs:
            The boolean parameters

    Returns:
        A dictionary with the proper values for each boolean
    """
    final_booleans = {}

    # Pure conv models (such as ConvNext) do not have `output_attentions`. If the signature has
    # `output_attentions`, it will be present here in `kwargs`, even if unset (in that case, as `None`)
    if "output_attentions" in kwargs:
        final_booleans["output_attentions"] = (
            kwargs["output_attentions"] if kwargs["output_attentions"] is not None else config.output_attentions
        )
    final_booleans["output_hidden_states"] = (
        kwargs["output_hidden_states"] if kwargs["output_hidden_states"] is not None else config.output_hidden_states
    )
    final_booleans["return_dict"] = kwargs["return_dict"] if kwargs["return_dict"] is not None else config.return_dict

    if "use_cache" in kwargs:
        final_booleans["use_cache"] = (
            kwargs["use_cache"] if kwargs["use_cache"] is not None else getattr(config, "use_cache", None)
        )


SUMMARY:
**Concise Summary: `booleans_processing` function**

This function processes input booleans of a running model. It takes a `config` object and additional keyword arguments (`kwargs`). The function returns a dictionary with the proper boolean values.

**Key Implementation Details:**

* Sets default values based on the `config` or `kwargs`
* Uses conditional statements to handle optional parameters
* Utilizes the `getattr` method to safely retrieve a value from the `config` object

**Notable Dependencies and Requirements:**

* `PretrainedConfig` class (part of the configuration object)
* No external dependencies listed

**Primary Purpose and Functionality:**

This function filters conditional parameters in the running model, providing default values when not provided. It ensures consistent processing across different models.


key: chroma:document
string_value:
CODE:

def unpack_inputs(func):
    """
    Decorator that processes the inputs to a Keras layer, passing them to the layer as keyword arguments. This enables
    downstream use of the inputs by their variable name, even if they arrive packed as a dictionary in the first input
    (common case in Keras).

    Args:
        func (`callable`):
            The callable function of the TensorFlow model.


    Returns:
        A callable that wraps the original `func` with the behavior described above.
    """

    original_signature = inspect.signature(func)

    @functools.wraps(func)
    def run_call_with_unpacked_inputs(self, *args, **kwargs):
        # isolates the actual `**kwargs` for the decorated function
        kwargs_call = {key: val for key, val in kwargs.items() if key not in dict(original_signature.parameters)}
        fn_args_and_kwargs = {key: val for key, val in kwargs.items() if key not in kwargs_call}
        fn_args_and_kwargs.update({"kwargs_call": kwargs_call})

        # move any arg into kwargs, if they exist
        fn_args_and_kwargs.update(dict(zip(func.__code__.co_varnames[1:], args)))

        # Encoder Decoder models delegate the application of the configuration options to their inner models.
        if "EncoderDecoder" in self.__class__.__name__:
            config = None
        else:
            config = self.config

        unpacked_inputs = input_processing(func, config, **fn_args_and_kwargs)
        return func(self, **unpacked_inputs)

    # Keras enforces the first layer argument to be passed, and checks it through `inspect.getfullargspec()`. This
    # function does not follow wrapper chains (i.e. ignores `functools.wraps()`), meaning that without the line below
    # Keras would attempt to check the first argument against the literal signature of the wrapper.
    run_call_with_unpacked_inputs.__signature__ = original_signature



SUMMARY:
**Summary**

`unpack_inputs` is a decorator that decouples Keras layers from packed input dictionaries. It:

1. **Unpacks inputs**: Extracts variable names and passes them to the layer via keyword arguments, preserving downstream usage.

2. **Handles nested structures**: Recognizes packed input dictionaries and unpacks their contents.

3. **Respects Keras constraints**: Ensures the first argument is passed without modification.

4. **Enables dynamic configuration**: Allows models like `EncoderDecoder` to delegate configuration options.

**Key aspects:**

* **Function signature manipulation**
* **Variable name extraction and passing**
* **Decorating with inherited interfaces (`run_call_with_unpacked_inputs.__signature__ = original_signature`)
* **Dependencies on `input_processing`, `functools`, `inspect`, and `keras` libraries**


key: chroma:document
string_value:
CODE:

def input_processing(func, config, **kwargs):
    """
    Process the input of each TensorFlow model including the booleans. In case of a list of symbolic inputs, each input
    has to be named accordingly to the parameters name, i.e. `input_ids = keras.Input(shape=(128,), dtype='int32',
    name="input_ids")` otherwise the order of the tensors will not be guaranteed during the training.

    Args:
        func (`callable`):
            The callable function of the TensorFlow model.
        config ([`PretrainedConfig`]):
            The config of the running model.
        **kwargs:
            The inputs of the model.

    Returns:
        Two lists, one for the missing layers, and another one for the unexpected layers.
    """
    signature = dict(inspect.signature(func).parameters)
    has_kwargs = bool(signature.pop("kwargs", None))
    signature.pop("self", None)
    parameter_names = list(signature.keys())
    main_input_name = parameter_names[0]
    main_input = kwargs.pop(main_input_name, None)
    output = {}
    allowed_types = (tf.Tensor, bool, int, ModelOutput, tuple, list, dict, np.ndarray)

    if "inputs" in kwargs["kwargs_call"]:
        warnings.warn(
            "The `inputs` argument is deprecated and will be removed in a future version, use `input_ids` instead.",
            FutureWarning,
        )

        output["input_ids"] = kwargs["kwargs_call"].pop("inputs")

    if "decoder_cached_states" in kwargs["kwargs_call"]:
        warnings.warn(
            "The `decoder_cached_states` argument is deprecated and will be removed in a future version, use"
            " `past_key_values` instead.",
            FutureWarning,
        )
        output["past_key_values"] = kwargs["kwargs_call"].pop("decoder_cached_states")

    if "past" in kwargs["kwargs_call"] and "past_key_values" in parameter_names:
        warnings.warn(
            "The `past` argument is deprecated and will be removed in a future version, use `past_key_values`"
            " instead.",
            FutureWarning,
        )
        kwargs["past_key_values"] = kwargs["kwargs_call"].pop("past")
    elif "past_key_values" in kwargs["kwargs_call"] and "past" in parameter_names:
        kwargs["past"] = kwargs["kwargs_call"].pop("past_key_values")

    if has_kwargs:
        output["kwargs"] = kwargs.pop("kwargs_call", {})
    else:
        if len(kwargs["kwargs_call"]) > 0:
            raise ValueError(
                "The following keyword arguments are not supported by this model:"
                f" {list(kwargs['kwargs_call'].keys())}."
            )
        kwargs.pop("kwargs_call")

    for k, v in kwargs.items():
        if isinstance(v, allowed_types) or tf.is_tensor(v) or v is None:
            output[k] = v
        else:
            raise ValueError(f"Data of type {type(v)} is not allowed only {allowed_types} is accepted for {k}.")

    if isinstance(main_input, (tuple, list)):
        for i, input in enumerate(main_input):
            # EagerTensors don't allow to use the .name property so we check for a real Tensor
            if is_tf_symbolic_tensor(input):
                # Tensor names have always the pattern `name:id` then we check only the
                # `name` part
                tensor_name = input.name.split(":")[0]

                if tensor_name in parameter_names:
                    output[tensor_name] = input
                else:
                    output[parameter_names[i]] = input
            elif isinstance(input, allowed_types) or input is None:
                output[parameter_names[i]] = input
            else:
                raise ValueError(
                    f"Data of type {type(input)} is not allowed only {allowed_types} is accepted for"
                    f" {parameter_names[i]}."
                )
    elif isinstance(main_input, Mapping):
        if "inputs" in main_input:
            warnings.warn(
                "The `inputs` argument is deprecated and will be removed in a future version, use `input_ids`"
                " instead.",
                FutureWarning,
            )

            output["input_ids"] = main_input.pop("inputs")

        if "decoder_cached_states" in main_input:
            warnings.warn(
                "The `decoder_cached_states` argument is deprecated and will be removed in a future version, use"
                " `past_key_values` instead.",
                FutureWarning,
            )
            output["past_key_values"] = main_input.pop("decoder_cached_states")

        for k, v in dict(main_input).items():
            if isinstance(v, allowed_types) or v is None:
                output[k] = v
            elif k not in parameter_names and "args" not in parameter_names:
                logger.warning(
                    f"The parameter {k} does not belongs to the parameter list {parameter_names} and will be ignored."
                )
                continue
            else:
                raise ValueError(f"Data of type {type(v)} is not allowed only {allowed_types} is accepted for {k}.")
    else:
        if tf.is_tensor(main_input) or main_input is None:
            output[main_input_name] = main_input
        else:
            raise ValueError(
                f"Data of type {type(main_input)} is not allowed only {allowed_types} is accepted for"
                f" {main_input_name}."
            )

    # Populates any unspecified argument with their default value, according to the signature.
    for name in parameter_names:
        if name not in list(output.keys()) and name != "args":
            output[name] = kwargs.pop(name, signature[name].default)

    # When creating a SavedModel TF calls the method with LayerCall.__call__(args, **kwargs)
    # So to respect the proper output we have to add this exception
    if "args" in output:
        if output["args"] is not None and is_tf_symbolic_tensor(output["args"]):
            tensor_name = output["args"].name.split(":")[0]
            output[tensor_name] = output["args"]
        else:
            # `args` in this case is always the first parameter, then `input_ids`
            output["input_ids"] = output["args"]

        del output["args"]

    if "kwargs" in output:
        del output["kwargs"]

    cast_output = {}
    for key, val in output.items():
        if isinstance(val, tf.Tensor) and val.dtype == tf.int64:
            cast_output[key] = tf.cast(val, tf.int32)
        elif isinstance(val, np.ndarray) and val.dtype == np.int64:
            cast_output[key] = val.astype(np.int32)
        else:
            cast_output[key] = val

    output = cast_output
    del cast_output

    if config is not None:
        boolean_dict = {
            k: v
            for k, v in output.items()
            if k in ["return_dict", "output_attentions", "output_hidden_states", "use_cache"]
        }

        output.update(
            booleans_processing(
                config=config,
                **boolean_dict,
            )
        )



SUMMARY:
**Summary:**

This is the `input_processing` function, which processes the input of each TensorFlow model. The main purpose is to validate and normalize the inputs according to the model's signature.

**Primary Purpose:**
The primary purpose is to process and validate the inputs to ensure they match the expected types and structures for the given model.

**Key Implementation Details:**

*   It uses the `inspect.signature` function to extract the model's parameters from the provided callable.
*   It validates the input types, preferring specific data types (e.g., tensors, ints) over others.
*   When encountering lists or tuples as inputs, it iterates through each element and assigns it to an output dictionary based on its name or index.

**Notable Dependencies and Requirements:**

*   The function relies on TensorFlow's `tf` namespace for tensor operations and the `inspect` module for signature inspection.
*   It assumes that the provided model is a callable that uses TensorFlow's API, with inputs validated against specific allowed types (e.g., tensors, ints).

**Returned Data Structure:**
The function returns a dictionary containing the normalized and validated inputs to the model.

```python
output = {
    # key-value pairs of the validated and normalized inputs
}
```

This summary captures the main functionality, processing rules, dependencies, and returned data structure of the `input_processing` function.


key: chroma:document
string_value:
CODE:

def dtype_byte_size(dtype):
    """
    Returns the size (in bytes) occupied by one parameter of type `dtype`.

    Example:

    ```py
    >>> dtype_byte_size(tf.float32)
    4
    ```
    """
    if dtype == tf.bool:
        return 1 / 8
    bit_search = re.search(r"[^\d](\d+)$", dtype.name)
    if bit_search is None:
        raise ValueError(f"`dtype` is not a valid dtype: {dtype}.")
    bit_size = int(bit_search.groups()[0])


SUMMARY:
**Summary**

The `dtype_byte_size` function calculates the size (in bytes) occupied by one instance of a given data type. It takes into account the types' native representation.

**Key Details:**

* Calculates byte size based on bitwise representation using regular expressions.
* Supports TensorFlow boolean types (`tf.bool`).
* Raises error if input `dtype` is not valid.

**Implementation:**

```python
def dtype_byte_size(dtype):
    """
    Returns the size (in bytes) occupied by one parameter of type `dtype`.

    Args:
        dtype (TensorFlow DataType): The data type to calculate byte size for.

    Returns:
        int: Size in bytes.
    Raises:
        ValueError: If input `dtype` is not valid.
    """
    if dtype == tf.bool:
        return 1 / 8
    bit_search = re.search(r"[^\d](\d+)$", dtype.name)
    if bit_search is None:
        raise ValueError(f"`dtype` is not a valid dtype: {dtype}.")
    bit_size = bit_search.groups()[0]
    return bit_size // 8
```


key: chroma:document
string_value:
CODE:

def strip_model_name_and_prefix(name, _prefix=None):
    if _prefix is not None and name.startswith(_prefix):
        name = name[len(_prefix) :]
        if name.startswith("/"):
            name = name[1:]
    if "model." not in name and len(name.split("/")) > 1:
        name = "/".join(name.split("/")[1:])


SUMMARY:
**Summary:**

`strip_model_name_and_prefix` function:
 
 * Takes a `name` parameter and an optional `_prefix`.
 * Strips the prefixed part of `name` if present, removing the prefix character-by-character and discarding any leading slashes.
 * Removes model-specific path segments from the resulting name if present (i.e., `name` is not in the format `model/model/subpath`).
 
 **Methods/Functions:**
 - `startswith`: used to check if `name` starts with `_prefix`.
 - `split()`, `join()` used on arrays of `/` substrings.
 
 **Interfaces:** None
 
 **External Dependencies:** None


key: chroma:document
string_value:
CODE:

def tf_shard_checkpoint(weights, max_shard_size="10GB", weights_name: str = TF2_WEIGHTS_NAME):
    """
    Splits a model state dictionary in sub-checkpoints so that the final size of each sub-checkpoint does not exceed a
    given size.

    The sub-checkpoints are determined by iterating through the `state_dict` in the order of its keys, so there is no
    optimization made to make each sub-checkpoint as close as possible to the maximum size passed. For example, if the
    limit is 10GB and we have weights of sizes [6GB, 6GB, 2GB, 6GB, 2GB, 2GB] they will get sharded as [6GB], [6+2GB],
    [6+2+2GB] and not [6+2+2GB], [6+2GB], [6GB].

    <Tip warning={true}>

    If one of the model's weight is bigger that `max_shard_size`, it will end up in its own sub-checkpoint which will
    have a size greater than `max_shard_size`.

    </Tip>

    Args:
        weights (`Dict[str, tf.RessourceVariable]`): The list of tf.RessourceVariable of a model to save.
        max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`):
            The maximum size of each sub-checkpoint. If expressed as a string, needs to be digits followed by a unit
            (like `"5MB"`).
    """
    max_shard_size = convert_file_size_to_int(max_shard_size)

    sharded_state_dicts = []
    current_block = []
    current_block_size = 0
    total_size = 0

    for item in weights:
        weight_size = item.numpy().size * dtype_byte_size(item.dtype)

        # If this weight is going to tip up over the maximal size, we split.
        if current_block_size + weight_size > max_shard_size:
            sharded_state_dicts.append(current_block)
            current_block = []
            current_block_size = 0

        current_block.append(item)
        current_block_size += weight_size
        total_size += weight_size

    # Add the last block
    sharded_state_dicts.append(current_block)

    # If we only have one shard, we return it
    if len(sharded_state_dicts) == 1:
        return {weights_name: sharded_state_dicts[0]}, None

    # Otherwise, let's build the index
    weight_map = {}
    shards = {}
    for idx, shard in enumerate(sharded_state_dicts):
        shard_file = weights_name.replace(".h5", f"-{idx+1:05d}-of-{len(sharded_state_dicts):05d}.h5")
        shard_file = shard_file.replace(
            ".safetensors", f"-{idx + 1:05d}-of-{len(sharded_state_dicts):05d}.safetensors"
        )
        shards[shard_file] = shard
        for weight in shard:
            weight_name = weight.name
            weight_map[weight_name] = shard_file

    # Add the metadata
    metadata = {"total_size": total_size}
    index = {"metadata": metadata, "weight_map": weight_map}


SUMMARY:
**Summary: `tf_shard_checkpoint` Function**

**Main Purpose and Functionality**
The `tf_shard_checkpoint` function splits a model state dictionary into sub-checkpoints to limit their total size. It iterates through the keys in the order they appear, grouping weights by key into contiguous chunks.

**Key Implementation Details**

* Iterates through the input `weights` dictionary and groups weights into chunks based on their size relative to `max_shard_size`.
* Assigns each chunk a unique file name with an index for easy identification.
* Returns two arrays: `shards`, containing the split weight dictionaries, and `index`, containing metadata and a mapping of original weight names to shard file names.

**Notable Dependencies or Requirements**

* `tf.RessourceVariable` and other TensorFlow types
* `dtype_byte_size` function (not shown in this code snippet)
* External dependencies:
	+ None explicitly mentioned, but `convert_file_size_to_int` function is assumed to be available


key: chroma:document
string_value:
CODE:

def load_tf_sharded_weights(model, shard_files, ignore_mismatched_sizes=False, strict=False, _prefix=None):
    """
    This is the same as `load_tf_weights` but for a sharded checkpoint. Detect missing and unexpected layers and load
    the TF weights from the shard file accordingly to their names and shapes.

    This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being
    loaded in the model.

    Args:
        model (`keras.models.Model`): The model in which to load the checkpoint.
        shard_files (`str` or `os.PathLike`): A list containing the sharded checkpoint names.
        ignore_mismatched_sizes`bool`, *optional`, defaults to `True`):
            Whether or not to ignore the mismatch between the sizes
        strict (`bool`, *optional*, defaults to `True`):
            Whether to strictly enforce that the keys in the model state dict match the keys in the sharded checkpoint.

    Returns:
        Three lists, one for the missing layers, another one for the unexpected layers, and a last one for the
        mismatched layers.
    """

    # Load the index
    unexpected_keys = set()
    saved_keys = set()
    mismatched_keys = set()

    # Since TF adds the name of the class to its weights, and uses the index and not the name of the layer to load
    # the weight, we have to get rid of the first prefix of the name of the layer.
    model_keys = set()
    model_layer_map = {}
    for i, k in enumerate(model.weights):
        layer_name = k.name
        if _prefix is not None and layer_name.startswith(_prefix):
            layer_name = layer_name[len(_prefix) :]
            layer_name = layer_name.lstrip("/")
        if not ("model." in layer_name or len(layer_name.split("/")) == 1):
            layer_name = "/".join(layer_name.split("/")[1:])
        model_keys.add(layer_name)
        model_layer_map[layer_name] = i

    for shard_file in shard_files:
        saved_weight_names_set, unexpected_keys_set, mismatched_keys_set = load_tf_shard(
            model,
            model_layer_map,
            shard_file,
            ignore_mismatched_sizes=ignore_mismatched_sizes,
            _prefix=_prefix,
        )
        saved_keys.update(saved_weight_names_set)
        unexpected_keys.update(unexpected_keys_set)
        mismatched_keys.update(mismatched_keys_set)
        gc.collect()

    missing_keys = model_keys - saved_keys
    if strict and (len(missing_keys) > 0 or len(unexpected_keys) > 0):
        error_message = f"Error(s) in loading state_dict for {model.__class__.__name__}"
        if len(missing_keys) > 0:
            str_missing_keys = ",".join([f'"{k}"' for k in missing_keys])
            error_message += f"\nMissing key(s): {str_missing_keys}."
        if len(unexpected_keys) > 0:
            str_unexpected_keys = ",".join([f'"{k}"' for k in unexpected_keys])
            error_message += f"\nMissing key(s): {str_unexpected_keys}."
        raise RuntimeError(error_message)



SUMMARY:
**Summary:**

This function, `load_tf_sharded_weights`, loads TF weights from sharded checkpoint files into a Keras model. It efficiently loads each checkpoint shard one by one in RAM and detects missing and unexpected layers.

**Main Purpose and Functionality:**
The main purpose of this function is to load optimized TF weights from sharded checkpoints into a Keras model, while detecting missing and unexpected layers.

**Key Implementation Details:**

*   Loads the `model` with its corresponding layer names and maps them to their indices.
*   Iterates through each sharded checkpoint file, loading the weights and updating the detected keys (missing, unexpected, mismatched).
*   Uses recursion (`load_tf_shard`) to load weights from each shard.

**Notable Dependencies and Requirements:**

*   `keras.models.Model`
*   `os.PathLike` for handling sharded checkpoint files
*   The `_prefix` parameter is used to extract layer names without the prefix added by TF

The function returns three lists: missing keys, unexpected keys, and mismatched keys.


key: chroma:document
string_value:
CODE:

def load_tf_shard(model, model_layer_map, resolved_archive_file, ignore_mismatched_sizes=False, _prefix=None):
    """
    Loads a shard from a sharded checkpoint file. Can be either H5 or Safetensors.
    Handles missing keys and unexpected keys.

    Args:
        model (`keras.models.Model`): Model in which the weights are loaded
        model_layer_map (`Dict`): A dictionary mapping the layer name to the index of the layer in the model.
        resolved_archive_file (`str`): Path to the checkpoint file from which the weights will be loaded
        ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`): Whether to ignore the mismatched keys

    Returns:
        `keras.models.Model`: Three lists, one for the layers that were found and succesfully restored (from the
        shard file), one for the mismatched layers, and another one for the unexpected layers.
    """
    saved_weight_names_set = set()
    saved_weights = {}
    mismatched_keys = set()
    unexpected_keys = set()
    # Read the H5 file
    try:
        with h5py.File(resolved_archive_file, "r") as sharded_checkpoint_file:
            # Retrieve the name of each layer from the H5 file
            saved_h5_model_layers_name = set(load_attributes_from_hdf5_group(sharded_checkpoint_file, "layer_names"))
            weight_value_tuples = []

            # Compute missing and unexpected sub layers
            # Store the weights in list of tuples that looks like [(weight_object, value_of_weight),...]
            for layer_name in saved_h5_model_layers_name:
                h5_layer_object = sharded_checkpoint_file[layer_name]
                saved_weights[layer_name] = np.asarray(h5_layer_object)

                saved_weight_names_set.add(layer_name)

                if layer_name not in model_layer_map:
                    unexpected_keys.add(layer_name)
                else:
                    symbolic_weight = model.weights[model_layer_map[layer_name]]

                    saved_weight_value = saved_weights[layer_name]
                    # If the current weight is found
                    if saved_weight_value is not None:
                        # Check if the shape of the current weight and the one from the H5 file are different
                        if K.int_shape(symbolic_weight) != saved_weight_value.shape:
                            # If yes we reshape the weight from the H5 file accordingly to the current weight
                            # If the two shapes are not compatible we raise an issue
                            try:
                                array = np.reshape(saved_weight_value, K.int_shape(symbolic_weight))
                            except ValueError as e:
                                if ignore_mismatched_sizes:
                                    mismatched_keys.add(
                                        (layer_name, saved_weight_value.shape, K.int_shape(symbolic_weight))
                                    )
                                    continue
                                else:
                                    raise e
                        else:
                            array = saved_weight_value

                    # We create the tuple that will be loaded and add it to the final list
                    weight_value_tuples.append((symbolic_weight, array))

        K.batch_set_value(weight_value_tuples)

        return saved_weight_names_set, unexpected_keys, mismatched_keys

    except Exception as e:
        try:
            with open(resolved_archive_file) as f:
                if f.read().startswith("version"):
                    raise OSError(
                        "You seem to have cloned a repository without having git-lfs installed. Please install "
                        "git-lfs and run `git lfs install` followed by `git lfs pull` in the folder "
                        "you cloned."
                    )
                else:
                    raise ValueError(
                        f"Unable to locate the file {resolved_archive_file} which is necessary to load this pretrained"
                        " model. Make sure you have saved the model properly."
                    ) from e
        except (UnicodeDecodeError, ValueError):
            raise OSError(
                f"Unable to load weights from TF checkpoint file for '{resolved_archive_file}' "
                f"at '{resolved_archive_file}'. "
                "If you tried to load a TF model from a sharded checkpoint, you should try converting the model "
                "by loading it in pytorch and saving it localy. A convertion script should be realeased soon."


SUMMARY:
**Summary:**

This function, `load_tf_shard`, loads weights from a sharded checkpoint file (either H5 or Safetensors) into a pre-trained Keras model. The provided code snippet encapsulates the main functionality of the `load_tf_shard` method.

### Overview

- **Primary purpose**: Load weights from a sharded checkpoint into a Keras model.
  
- **Key algorithms and design choices**:
    - Handles missing and unexpected layer names by storing them in separate sets for later consideration.
    - Reshapes weights with mismatched shapes using `np.reshape`.
    - Uses `K.batch_set_value` to update the model's weights.

### Notable dependencies:

*   `keras.models.Model`
*   `Dict`
*   `H5Py` or `Safetensors` for loading the checkpoint file
*   `numpy` (`np`) and `pydot` (`K`) for numerical computations

### Notes:

- The function attempts to load weights from various files without Git LFS. If no such check succeeds, it fails with an exception.
  
```markdown
# Function Summary

## Function name: load_tf_shard

## Type: Load function

## Parameters:
### model (keras.models.Model)
Model in which the weights are loaded
### model_layer_map (Dict)
A dictionary mapping the layer name to the index of the layer in the model.
### resolved_archive_file (str)
Path to the checkpoint file from which the weights will be loaded
### ignore_mismatched_sizes (bool, optional, defaults to False)
Whether to ignore the mismatched keys

## Returns:
list: 
1.   Three lists: One for found layers, one for mismatched layers and another for unexpected layers.
```

**Functionality in 2 Lines:**

The function `load_tf_shard` maps layer names from a sharded checkpoint file to their corresponding indices in the model. If found, it stores these weights; otherwise, it identifies them as mismatched or unexpected.

(Note that there has not been enough output provided for all of these information, please let me know if I can add anything.)


key: chroma:document
string_value:
CODE:

def load_tf_sharded_weights_from_safetensors(
    model, shard_files, ignore_mismatched_sizes=False, strict=False, _prefix=None
):
    """
    This is the same as `load_tf_weights_from_safetensors` but for a sharded TF-format safetensors checkpoint.
    Detect missing and unexpected layers and load the TF weights from the shard file accordingly to their names and
    shapes.

    This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being
    loaded in the model.

    Args:
        model (`keras.models.Model`): The model in which to load the checkpoint.
        shard_files (`str` or `os.PathLike`): A list containing the sharded checkpoint names.
        ignore_mismatched_sizes`bool`, *optional`, defaults to `True`):
            Whether or not to ignore the mismatch between the sizes
        strict (`bool`, *optional*, defaults to `True`):
            Whether to strictly enforce that the keys in the model state dict match the keys in the sharded checkpoint.

    Returns:
        Three lists, one for the missing layers, another one for the unexpected layers, and a last one for the
        mismatched layers.
    """

    # Load the index
    unexpected_keys = set()
    all_missing_keys = []
    mismatched_keys = set()

    for shard_file in shard_files:
        missing_layers, unexpected_layers, mismatched_layers = load_tf_weights_from_safetensors(
            model,
            shard_file,
            ignore_mismatched_sizes=ignore_mismatched_sizes,
            _prefix=_prefix,
        )
        all_missing_keys.append(set(missing_layers))
        unexpected_keys.update(unexpected_layers)
        mismatched_keys.update(mismatched_layers)
        gc.collect()
    missing_keys = set.intersection(*all_missing_keys)

    if strict and (len(missing_keys) > 0 or len(unexpected_keys) > 0):
        error_message = f"Error(s) in loading state_dict for {model.__class__.__name__}"
        if len(missing_keys) > 0:
            str_missing_keys = ",".join([f'"{k}"' for k in missing_keys])
            error_message += f"\nMissing key(s): {str_missing_keys}."
        if len(unexpected_keys) > 0:
            str_unexpected_keys = ",".join([f'"{k}"' for k in unexpected_keys])
            error_message += f"\nMissing key(s): {str_unexpected_keys}."
        raise RuntimeError(error_message)



SUMMARY:
**Summary of `load_tf_sharded_weights_from_safetensors` Function**

### Purpose and Functionality

This function loads TF-weights from sharded safetensors checkpoint into a given Keras model. It efficiently loads each shard in RAM, detects missing and unexpected layers, and raises errors if strict enforcement is required.

### Key Implementation Details

* Uses `load_tf_weights_from_safetensors` with slight modifications for sharding.
* Returns three lists: missing keys, unexpected keys, and mismatched keys.
* Enforces strict loading if specific conditions are met.

### Notable Dependencies and Requirements

* Keras models
* Safetensors library
* Python's garbage collector ( gc.collect() )

### Method Signatures and Interfaces

* `load_tf_weights_from_safetensors`: a companion function with similar signature.


key: chroma:document
string_value:
CODE:

def load_tf_weights(model, resolved_archive_file, ignore_mismatched_sizes=False, _prefix=None):
    """
    Detect missing and unexpected layers and load the TF weights from the shard file accordingly to their names and
    shapes.

    Args:
        model (`keras.models.Model`):
            The model to load the weights into.
        resolved_archive_file (`str`):
            The location of the H5 file.
        ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`):
            Whether or not to ignore weights with shapes that don't match between the checkpoint of the model.

    Returns:
        Three lists, one for the missing layers, another one for the unexpected layers, and a last one for the
        mismatched layers.
    """
    if resolved_archive_file.endswith(".safetensors"):
        load_function = load_tf_weights_from_safetensors
    else:
        load_function = load_tf_weights_from_h5

    return load_function(
        model, resolved_archive_file, ignore_mismatched_sizes=ignore_mismatched_sizes, _prefix=_prefix


SUMMARY:
**Function Summary: `load_tf_weights`**

**Purpose:** This function loads TensorFlow (TF) weights into a Keras model.

**Key Functionality:**

* Detects missing and unexpected layers in the model.
* Loads TF weights from an H5 file, adhering to layer names and shapes.
* Provides three lists of layers: missing, unexpected, and mismatched.

**Implementation Details:**

* The function determines whether to use `load_tf_weights_from_safetensors` or `load_tf_weights_from_h5` based on the input archive file extension.
* It takes an optional `_prefix` argument to modify the loaded weight names.

**External Dependencies:**

* Kerass models (required for model instance)
* Safentensors (optional dependency)

**Method Signatures:**

* `load_tf_weights_from_safetensors(model, resolved_archive_file, ignore_mismatched_sizes=False, _prefix=None)`.
* `load_tf_weights_from_h5(model, resolved_archive_file, ignore_mismatched_sizes=False, _prefix=None)`.


key: chroma:document
string_value:
CODE:

def load_tf_weights_from_h5(model, resolved_archive_file, ignore_mismatched_sizes=False, _prefix=None):
    mismatched_layers = []

    # Read the H5 file
    with h5py.File(resolved_archive_file, "r") as sharded_checkpoint_file:
        # Retrieve the name of each layer from the H5 file
        saved_h5_model_layers_name = set(load_attributes_from_hdf5_group(sharded_checkpoint_file, "layer_names"))

        # Find the missing layers from the high level list of layers
        missing_layers = list({layer.name for layer in model.layers} - saved_h5_model_layers_name)

        # Find the unexpected layers from the high level list of layers
        unexpected_layers = list(saved_h5_model_layers_name - {layer.name for layer in model.layers})
        saved_weight_names_set = set()
        symbolic_weights_names = set()
        weight_value_tuples = []

        # Compute missing and unexpected sub layers
        # Store the weights in list of tuples that looks like [(weight_object, value_of_weight),...]
        for layer in model.layers:
            # if layer_name from the H5 file belongs to the layers from the instantiated model
            if layer.name in saved_h5_model_layers_name:
                # Get the H5 layer object from its name
                h5_layer_object = sharded_checkpoint_file[layer.name]
                # Get all the weights as a list from the layer object
                symbolic_weights = layer.trainable_weights + layer.non_trainable_weights
                saved_weights = {}

                # Create a dict from the H5 saved model that looks like {"weight_name": weight_value}
                # And a set with only the names
                for weight_name in load_attributes_from_hdf5_group(h5_layer_object, "weight_names"):
                    # TF names always start with the model name so we ignore it
                    name = "/".join(weight_name.split("/")[1:])

                    if _prefix is not None:
                        name = _prefix + "/" + name

                    saved_weights[name] = np.asarray(h5_layer_object[weight_name])

                    # Add the updated name to the final list for computing missing/unexpected values
                    saved_weight_names_set.add(name)

                # Loop over each weights from the instantiated model and compare with the weights from the H5 file
                for symbolic_weight in symbolic_weights:
                    # TF names always start with the model name so we ignore it
                    if _prefix is not None:
                        delimeter = len(_prefix.split("/"))
                        symbolic_weight_name = "/".join(
                            symbolic_weight.name.split("/")[:delimeter]
                            + symbolic_weight.name.split("/")[delimeter + 1 :]
                        )
                    else:
                        symbolic_weight_name = "/".join(symbolic_weight.name.split("/")[1:])

                    # here we check if the current weight is among the weights from the H5 file
                    # If yes, get the weight_value of the corresponding weight from the H5 file
                    # If not, make the value to None
                    saved_weight_value = saved_weights.get(symbolic_weight_name, None)

                    # Retrocompatibility patch: some embeddings are stored with the weights name (e.g. Bart's
                    # `model.shared/embeddings:0` are stored as `model.shared/weights:0`)
                    if saved_weight_value is None and symbolic_weight_name.endswith("embeddings:0"):
                        symbolic_weight_name = symbolic_weight_name[:-12] + "weight:0"
                        saved_weight_value = saved_weights.get(symbolic_weight_name, None)

                    # Add the updated name to the final list for computing missing/unexpected values
                    symbolic_weights_names.add(symbolic_weight_name)

                    # If the current weight is found
                    if saved_weight_value is not None:
                        # Check if the shape of the current weight and the one from the H5 file are different
                        if K.int_shape(symbolic_weight) != saved_weight_value.shape:
                            # If yes we reshape the weight from the H5 file accordingly to the current weight
                            # If the two shapes are not compatible we raise an issue
                            try:
                                array = np.reshape(saved_weight_value, K.int_shape(symbolic_weight))
                            except ValueError as e:
                                if ignore_mismatched_sizes:
                                    mismatched_layers.append(
                                        (symbolic_weight_name, saved_weight_value.shape, K.int_shape(symbolic_weight))
                                    )
                                    continue
                                else:
                                    raise e
                        else:
                            array = saved_weight_value

                        # We create the tuple that will be loaded and add it to the final list
                        weight_value_tuples.append((symbolic_weight, array))

    # Load all the weights
    K.batch_set_value(weight_value_tuples)

    # Compute the missing and unexpected layers
    missing_layers.extend(list(symbolic_weights_names - saved_weight_names_set))
    unexpected_layers.extend(list(saved_weight_names_set - symbolic_weights_names))



SUMMARY:
**Summary: Load TF Weights from H5 File**

This function loads weights from a `.h5` file into a Keras model. It:

1. **Loads Model Layers**: Retrieves the name of each layer from the H5 file and compares them with the layers in the instantiated model.
2. **Compares Weights**: Loops through the weights of the model, compares them with the weights from the H5 file, and loads missing or unexpected weights.
3. **Reshapes Missing Weights**: Reshapes the weight from the H5 file to match the shape of the corresponding weight in the model if necessary.
4. **Uploads Loaded Weights**: Sets the value of the loaded weights using `K.batch_set_value`.
5. **Returns Mismatched Layers**: Returns three lists: mismatched layers, missing layers, and unexpected layers.

**Key Implementation Details**

* Uses `h5py` to read the H5 file.
* Compares weights using `load_attributes_from_hdf5_group` function.
* Reshapes missing weights using `np.reshape`.
* Uploads loaded weights using `K.batch_set_value`.

**Notable Dependencies/Requirements**

* Keras
* H5Py
* NumPy (for numpy arrays and manipulation)

**Function Signature**
```python
def load_tf_weights_from_h5(model: object, resolved_archive_file: str, ignore_mismatched_sizes: bool = False, _prefix: str = None) -> (list, list, list):
    ...
```
This function takes a Keras model, the path to the H5 file, and optional parameters for ignoring mismatched sizes and prefix. It returns three lists: mismatched layers, missing layers, and unexpected layers.


key: chroma:document
string_value:
CODE:

def load_tf_weights_from_safetensors(model, resolved_archive_file, ignore_mismatched_sizes=False, _prefix=None):
    # Read the safetensors file
    with safe_open(resolved_archive_file, framework="tf") as safetensors_archive:
        mismatched_layers = []
        weight_names = [strip_model_name_and_prefix(w.name, _prefix=_prefix) for w in model.weights]
        loaded_weight_names = list(safetensors_archive.keys())
        # Find the missing layers from the high level list of layers
        missing_layers = list(set(weight_names) - set(loaded_weight_names))
        # Find the unexpected layers from the high level list of layers
        unexpected_layers = list(set(loaded_weight_names) - set(weight_names))

        for weight in model.weights:
            weight_name = strip_model_name_and_prefix(weight.name, _prefix=_prefix)
            if weight_name in loaded_weight_names:
                weight_value = safetensors_archive.get_tensor(weight_name)
                # Check if the shape of the current weight and the one from the H5 file are different
                if K.int_shape(weight) != weight_value.shape:
                    # If yes we reshape the weight from the H5 file accordingly to the current weight
                    # If the two shapes are not compatible we raise an issue
                    try:
                        weight_value = tf.reshape(weight_value, K.int_shape(weight))
                    except (ValueError, tf.errors.InvalidArgumentError) as e:
                        if ignore_mismatched_sizes:
                            mismatched_layers.append((weight_name, weight_value.shape, K.int_shape(weight)))
                            continue
                        else:
                            raise e

                K.set_value(weight, weight_value)  # weight.assign() might break if weight is a DTensor


SUMMARY:
**Function Summary: `load_tf_weights_from_safetensors`**

 Loads TF weights from SAFETENSORS archives for a given TensorFlow model.

### Purpose and Functionality

Loads TF weights from a SAFETENSORS archive while handling potential shape mismatches between loaded and saved weights. Supports ignoring size mismatches in such cases.

**Key Implementation Details:**

*   Utilizes `safe_open` to open the SAFETENSORS archive.
*   Extracts weight names using `strip_model_name_and_prefix`.
*   Compares and matches loaded vs saved weight values for each layer, handling shape mismatches via reshaping.
*   Returns missing, unexpected, and mismatched layers.

**Notable Dependencies:**

*   `tf.keras`: TensorFlow Keras API
*   `keras.utils`: Additional utility functions

**Assumptions:**

*   Model is a TensorFlow model where weights are stored in the `$model.weights` attribute.
*   SAFETENSORS archive corresponds to a Tensorflow framework using the "tf" configuration.


key: chroma:document
string_value:
CODE:

def init_copy_embeddings(old_embeddings, new_num_tokens):
    r"""
    This function aims to reduce the embeddings in case new_num_tokens < old_num_tokens or to pad with -1 in case
    new_num_tokens > old_num_tokens. A mask is also computed in order to know which weight in the embeddings should be
    kept or not. Example:

        - if new_num_tokens=5 and old_num_tokens=4 and old_embeddings=[w1,w2,w3,w4]

            -  mask=[True,True,True,True,False] and current_weights=[w1,w2,w3,w4,-1]
        - if new_num_tokens=4 and old_num_tokens=5 and old_embeddings=[w1,w2,w3,w4,w5]

            - mask=[True,True,True,True] and current_weights=[w1,w2,w3,w4]
    """
    old_num_tokens, old_embedding_dim = shape_list(old_embeddings)
    size_diff = new_num_tokens - old_num_tokens

    # initialize new embeddings
    # Copy token embeddings from the previous ones
    if tf.math.greater(size_diff, 0):
        # if the new size is greater than the old one, we extend the current embeddings with a padding until getting new size
        # and we create a mask to properly identify the padded values and be replaced by the values of the newly created
        # embeddings
        current_weights = tf.pad(
            old_embeddings.value(), tf.convert_to_tensor([[0, size_diff], [0, 0]]), constant_values=-1
        )
        num_tokens_to_copy = min(old_num_tokens, new_num_tokens)
        mask = tf.fill(tf.convert_to_tensor([num_tokens_to_copy, 1]), True)
        mask = tf.pad(mask, tf.convert_to_tensor([[0, size_diff], [0, 0]]), constant_values=False)
    else:
        # if the new size if lower than the old one, we take the current embeddings until the new size
        current_weights = tf.slice(
            old_embeddings.value(),
            tf.convert_to_tensor([0, 0]),
            tf.convert_to_tensor([new_num_tokens, old_embedding_dim]),
        )
        mask = tf.fill(tf.convert_to_tensor([new_num_tokens, 1]), True)



SUMMARY:
**Function Summary: `init_copy_embeddings`**

Purpose:
This function reduces or pads embeddings based on the new number of tokens compared to the old number. It computes a mask to identify which weights should be kept or replaced with padded values.

Implementation Details:

* If `new_num_tokens` is greater than `old_num_tokens`, it pads the old embeddings with -1 and creates a mask.
* If `new_num_tokens` is less than or equal to `old_num_tokens`, it takes a slice of the old embeddings and computes a mask.

Key Features:

* Uses TensorFlow padding and slicing operations
* Utilizes a mask to identify which weights should be kept or replaced

Notable Functions/Classes:

* `tf.pad`
* `tf.slice`

Dependencies:

* TensorFlow
* shape_list (not shown in the code snippet)


key: chroma:document
string_value:
CODE:

class TFPreTrainedModel(keras.Model, TFModelUtilsMixin, TFGenerationMixin, PushToHubMixin):
    r"""
    Base class for all TF models.

    [`TFPreTrainedModel`] takes care of storing the configuration of the models and handles methods for loading,
    downloading and saving models as well as a few methods common to all models to:

        - resize the input embeddings,
        - prune heads in the self-attention heads.

    Class attributes (overridden by derived classes):

        - **config_class** ([`PretrainedConfig`]) -- A subclass of [`PretrainedConfig`] to use as configuration class
          for this model architecture.
        - **base_model_prefix** (`str`) -- A string indicating the attribute associated to the base model in derived
          classes of the same architecture adding modules on top of the base model.
        - **main_input_name** (`str`) -- The name of the principal input to the model (often `input_ids` for NLP
          models, `pixel_values` for vision models and `input_values` for speech models).
    """

    config_class = None
    base_model_prefix = ""
    main_input_name = "input_ids"
    _auto_class = None
    _using_dummy_loss = None
    _label_to_output_map = None

    # a list of re pattern of tensor names to ignore from the model when loading the model weights
    # (and avoid unnecessary warnings).
    _keys_to_ignore_on_load_missing = None
    # a list of re pattern of tensor names to ignore from the weights when loading the model weights
    # (and avoid unnecessary warnings).
    _keys_to_ignore_on_load_unexpected = None
    _requires_load_weight_prefix = False

    @property
    def dummy_inputs(self) -> Dict[str, tf.Tensor]:
        """
        Dummy inputs to build the network.

        Returns:
            `Dict[str, tf.Tensor]`: The dummy inputs.
        """
        dummies = {}
        for key, spec in self.input_signature.items():
            # 2 is the most correct arbitrary size. I will not be taking questions
            dummy_shape = [dim if dim is not None else 2 for dim in spec.shape]
            if spec.shape[0] is None:
                # But let's make the batch size 1 to save memory anyway
                dummy_shape[0] = 1
            dummies[key] = tf.ones(shape=dummy_shape, dtype=spec.dtype)
            if key == "token_type_ids":
                # Some models have token_type_ids but with a vocab_size of 1
                dummies[key] = tf.zeros_like(dummies[key])
        if self.config.add_cross_attention and "encoder_hidden_states" in inspect.signature(self.call).parameters:
            if "encoder_hidden_states" not in dummies:
                if self.main_input_name == "input_ids":
                    dummies["encoder_hidden_states"] = tf.ones(
                        shape=(1, 2, self.config.hidden_size), dtype=tf.float32, name="encoder_hidden_states"
                    )
                else:
                    raise NotImplementedError(
                        "Model has cross-attention but we couldn't infer the shape for the encoder hidden states. Please manually override dummy_inputs!"
                    )
        return dummies

    def build_in_name_scope(self):
        with tf.name_scope(self.name):
            self.build(input_shape=None)

    @property
    def framework(self) -> str:
        """
        :str: Identifies that this is a TensorFlow model.
        """
        return "tf"

    def build(self, input_shape=None):
        pass  # This is just here to make sure we don't call the superclass build()

    def __init__(self, config, *inputs, **kwargs):
        super().__init__(*inputs, **kwargs)
        if not isinstance(config, PretrainedConfig):
            raise TypeError(
                f"Parameter config in `{self.__class__.__name__}(config)` should be an instance of class "
                "`PretrainedConfig`. To create a model from a pretrained model use "
                f"`model = {self.__class__.__name__}.from_pretrained(PRETRAINED_MODEL_NAME)`"
            )
        # Save config and origin of the pretrained weights if given in model
        self.config = config
        self.name_or_path = config.name_or_path
        self.generation_config = GenerationConfig.from_model_config(config) if self.can_generate() else None
        self._set_save_spec(self.input_signature)

    def get_config(self):
        return self.config.to_dict()

    @functools.wraps(keras.Model.fit)
    def fit(self, *args, **kwargs):
        args, kwargs = convert_batch_encoding(*args, **kwargs)
        return super().fit(*args, **kwargs)

    @functools.wraps(keras.Model.train_on_batch)
    def train_on_batch(self, *args, **kwargs):
        args, kwargs = convert_batch_encoding(*args, **kwargs)
        return super().train_on_batch(*args, **kwargs)

    @functools.wraps(keras.Model.test_on_batch)
    def test_on_batch(self, *args, **kwargs):
        args, kwargs = convert_batch_encoding(*args, **kwargs)
        return super().test_on_batch(*args, **kwargs)

    @functools.wraps(keras.Model.predict_on_batch)
    def predict_on_batch(self, *args, **kwargs):
        args, kwargs = convert_batch_encoding(*args, **kwargs)
        return super().predict_on_batch(*args, **kwargs)

    @functools.wraps(keras.Model.predict)
    def predict(self, *args, **kwargs):
        args, kwargs = convert_batch_encoding(*args, **kwargs)
        return super().predict(*args, **kwargs)

    @functools.wraps(keras.Model.evaluate)
    def evaluate(self, *args, **kwargs):
        args, kwargs = convert_batch_encoding(*args, **kwargs)
        return super().evaluate(*args, **kwargs)

    @classmethod
    def from_config(cls, config, **kwargs):
        if isinstance(config, PretrainedConfig):
            return cls._from_config(config, **kwargs)
        return cls._from_config(cls.config_class.from_dict(config, **kwargs))

    @classmethod
    def _from_config(cls, config, **kwargs):
        """
        All context managers that the model should be initialized under go here.
        """
        return cls(config, **kwargs)

    def get_head_mask(self, head_mask: tf.Tensor | None, num_hidden_layers: int) -> tf.Tensor:
        """
        Prepare the head mask if needed.

        Args:
            head_mask (`tf.Tensor` with shape `[num_heads]` or `[num_hidden_layers x num_heads]`, *optional*):
                The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard).
            num_hidden_layers (`int`):
                The number of hidden layers in the model.

        Returns:
            `tf.Tensor` with shape `[num_hidden_layers x batch x num_heads x seq_length x seq_length]` or list with
            `[None]` for each layer.
        """
        if head_mask is not None:
            head_mask = self._convert_head_mask_to_5d(head_mask, num_hidden_layers)
        else:
            head_mask = [None] * num_hidden_layers

        return head_mask

    def _convert_head_mask_to_5d(self, head_mask, num_hidden_layers):
        """-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]"""
        if head_mask.shape.rank == 1:
            head_mask = head_mask[None, None, :, None, None]
            head_mask = tf.repeat(head_mask, repeats=num_hidden_layers, axis=0)
        elif head_mask.shape.rank == 2:
            head_mask = head_mask[:, None, :, None, None]
        assert head_mask.shape.rank == 5, f"head_mask.dim != 5, instead {head_mask.dim()}"
        head_mask = tf.cast(head_mask, tf.float32)  # switch to float if need + fp16 compatibility
        return head_mask

    @tf.function
    def serving(self, inputs):
        """
        Args:
        Method used for serving the model. Does not have a specific signature, but will be specialized as concrete
        functions when saving with `save_pretrained`.
            inputs (`Dict[str, tf.Tensor]`):
                The input of the saved model as a dictionary of tensors.
        """
        output = self.call(inputs)

        return self.serving_output(output)

    @property
    def input_signature(self) -> Dict[str, tf.TensorSpec]:
        """
        This property should return a dict mapping input names to tf.TensorSpec objects, representing the expected
        shape and dtype for model inputs. It is used for both serving and for generating dummy inputs.
        """
        model_inputs = list(inspect.signature(self.call).parameters)
        sig = {}
        if "input_ids" in model_inputs:
            if self.__class__.__name__.endswith("ForMultipleChoice"):
                text_dims = 3
            else:
                text_dims = 2
            for input_name in (
                "input_ids",
                "attention_mask",
                "token_type_ids",
                "decoder_input_ids",
                "decoder_attention_mask",
            ):
                if input_name in model_inputs:
                    sig[input_name] = tf.TensorSpec([None] * text_dims, tf.int32, name=input_name)
        if "pixel_values" in model_inputs:
            pixel_values_shape = [None, None, None, None]
            if hasattr(self.config, "vision_config"):
                vision_config = self.config.vision_config
            else:
                vision_config = self.config
            if hasattr(vision_config, "num_channels"):
                pixel_values_shape[1] = vision_config.num_channels
            else:
                raise NotImplementedError(
                    "Could not infer number of channels from config, please override input_signature to specify input shapes."
                )
            if hasattr(vision_config, "image_size"):
                pixel_values_shape[2] = pixel_values_shape[3] = vision_config.image_size
            elif hasattr(vision_config, "input_size"):
                pixel_values_shape[2] = pixel_values_shape[3] = vision_config.input_size
            else:
                raise NotImplementedError(
                    "Could not infer input image shape from config, please override input_signature to specify input shapes."
                )
            sig["pixel_values"] = tf.TensorSpec(pixel_values_shape, tf.float32, name="pixel_values")
        if "input_features" in model_inputs:
            raise NotImplementedError("Audio models need a manually defined input_signature")
        return sig

    def serving_output(self, output):
        """
        Prepare the output of the saved model. Can be overridden if specific serving modifications are required.
        """
        if not isinstance(output, ModelOutput):
            return output
        for key in output:
            if key.endswith("hidden_states") and not getattr(self.config, "output_hidden_states", False):
                output[key] = None
            elif key.endswith("attentions") and not getattr(self.config, "output_attentions", False):
                output[key] = None
            elif key == "past_key_values" and not getattr(self.config, "use_cache", False):
                output[key] = None
            elif key == "cross_attentions" and not (
                getattr(self.config, "output_attentions", False) and getattr(self.config, "add_cross_attention", False)
            ):
                output[key] = None
            if isinstance(output[key], (tuple, list)):
                try:
                    output[key] = tf.convert_to_tensor(output[key])
                except (ValueError, tf.errors.InvalidArgumentError):
                    pass  # Layers may not have the same dimensions
        return output

    @classmethod
    def can_generate(cls) -> bool:
        """
        Returns whether this model can generate sequences with `.generate()`.

        Returns:
            `bool`: Whether this model can generate sequences with `.generate()`.
        """
        # Detects whether `prepare_inputs_for_generation` has been overwritten, which is a requirement for generation.
        # Alternativelly, the model can also have a custom `generate` function.
        if "GenerationMixin" in str(cls.prepare_inputs_for_generation) and "GenerationMixin" in str(cls.generate):
            return False
        return True

    def get_input_embeddings(self) -> keras.layers.Layer:
        """
        Returns the model's input embeddings layer.

        Returns:
            `tf.Variable`: The embeddings layer mapping vocabulary to hidden states.
        """
        main_layer = getattr(self, self.base_model_prefix, self)

        if main_layer is not self:
            return main_layer.get_input_embeddings()
        else:
            raise NotImplementedError

    def _save_checkpoint(self, checkpoint_dir, epoch):
        if not os.path.isdir(checkpoint_dir):
            os.mkdir(checkpoint_dir)
        # We avoid tf.train.checkpoint or saving weights in TF format, even though that includes optimizer
        # state for us, because it requires special handling for objects like custom losses, which we use
        # internally and which users are likely to use too
        weights_path = os.path.join(checkpoint_dir, "weights.h5")
        self.save_weights(weights_path)
        extra_data = {"epoch": epoch, "optimizer_state": self.optimizer.get_weights()}
        extra_data_path = os.path.join(checkpoint_dir, "extra_data.pickle")
        with open(extra_data_path, "wb") as f:
            pickle.dump(extra_data, f)

    def prepare_tf_dataset(
        self,
        dataset: "datasets.Dataset",  # noqa:F821
        batch_size: int = 8,
        shuffle: bool = True,
        tokenizer: Optional["PreTrainedTokenizerBase"] = None,
        collate_fn: Optional[Callable] = None,
        collate_fn_args: Optional[Dict[str, Any]] = None,
        drop_remainder: Optional[bool] = None,
        prefetch: bool = True,
    ):
        """
        Wraps a HuggingFace [`~datasets.Dataset`] as a `tf.data.Dataset` with collation and batching. This method is
        designed to create a "ready-to-use" dataset that can be passed directly to Keras methods like `fit()` without
        further modification. The method will drop columns from the dataset if they don't match input names for the
        model. If you want to specify the column names to return rather than using the names that match this model, we
        recommend using `Dataset.to_tf_dataset()` instead.

        Args:
            dataset (`Any`):
                A [~`datasets.Dataset`] to be wrapped as a `tf.data.Dataset`.
            batch_size (`int`, *optional*, defaults to 8):
                The size of batches to return.
            shuffle (`bool`, defaults to `True`):
                Whether to return samples from the dataset in random order. Usually `True` for training datasets and
                `False` for validation/test datasets.
            tokenizer ([`PreTrainedTokenizerBase`], *optional*):
                A `PreTrainedTokenizer` that will be used to pad samples to create batches. Has no effect if a specific
                `collate_fn` is passed instead.
            collate_fn (`Callable`, *optional*):
                A function that collates samples from the dataset into a single batch. Defaults to
                `DefaultDataCollator` if no `tokenizer` is supplied or `DataCollatorWithPadding` if a `tokenizer` is
                passed.
            collate_fn_args (`Dict[str, Any]`, *optional*):
                A dict of arguments to pass to the `collate_fn` alongside the list of samples.
            drop_remainder (`bool`, *optional*):
                Whether to drop the final batch, if the batch_size does not evenly divide the dataset length. Defaults
                to the same setting as `shuffle`.
            prefetch (`bool`, defaults to `True`):
                Whether to add prefetching to the end of the `tf.data` pipeline. This is almost always beneficial for
                performance, but can be disabled in edge cases.


        Returns:
            `Dataset`: A `tf.data.Dataset` which is ready to pass to the Keras API.
        """
        requires_backends(self, ["datasets"])
        import datasets

        if collate_fn is None:
            if tokenizer is None:
                collate_fn = DefaultDataCollator(return_tensors="np")
            else:
                collate_fn = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="np")
        if collate_fn_args is None:
            collate_fn_args = {}

        if not isinstance(dataset, datasets.Dataset):
            raise TypeError("Dataset argument should be a datasets.Dataset!")
        model_inputs = list(inspect.signature(self.call).parameters)
        model_labels = find_labels(self.__class__)
        if "cols_to_retain" in list(inspect.signature(dataset._get_output_signature).parameters.keys()):
            output_signature, _ = dataset._get_output_signature(
                dataset,
                batch_size=None,
                collate_fn=collate_fn,
                collate_fn_args=collate_fn_args,
                cols_to_retain=model_inputs,
            )
        else:
            # TODO Matt: This is a workaround for older versions of datasets that are missing the `cols_to_retain`
            #            argument. We should remove this once the minimum supported version of datasets is > 2.3.2
            unwanted_columns = [
                feature
                for feature in dataset.features
                if feature not in model_inputs and feature not in ("label_ids", "label")
            ]
            dataset = dataset.remove_columns(unwanted_columns)
            output_signature, _ = dataset._get_output_signature(
                dataset, batch_size=None, collate_fn=collate_fn, collate_fn_args=collate_fn_args
            )
        output_columns = list(output_signature.keys())
        feature_cols = [col for col in output_columns if col in model_inputs and col not in model_labels]
        label_cols = [col for col in output_columns if col in model_labels]

        # Backwards compatibility for older versions of datasets. Previously, if `columns` or `label_cols`
        # were a single element list, the returned element spec would be a single element. Now, passing [feature]
        # will return a dict structure {"feature": feature}, and passing a single string will return a single element.
        feature_cols = feature_cols[0] if len(feature_cols) == 1 else feature_cols
        label_cols = label_cols[0] if len(label_cols) == 1 else label_cols

        if drop_remainder is None:
            drop_remainder = shuffle
        tf_dataset = dataset.to_tf_dataset(
            columns=feature_cols,
            label_cols=label_cols,
            batch_size=batch_size,
            shuffle=shuffle,
            drop_remainder=drop_remainder,
            collate_fn=collate_fn,
            collate_fn_args=collate_fn_args,
            prefetch=prefetch,
        )
        return tf_dataset

    def compile(
        self,
        optimizer="rmsprop",
        loss="auto_with_warning",
        metrics=None,
        loss_weights=None,
        weighted_metrics=None,
        run_eagerly=None,
        steps_per_execution=None,
        **kwargs,
    ):
        """
        This is a thin wrapper that sets the model's loss output head as the loss if the user does not specify a loss
        function themselves.
        """
        if loss in ("auto_with_warning", "passthrough"):  # "passthrough" for workflow backward compatibility
            logger.info(
                "No loss specified in compile() - the model's internal loss computation will be used as the "
                "loss. Don't panic - this is a common way to train TensorFlow models in Transformers! "
                "To disable this behaviour please pass a loss argument, or explicitly pass "
                "`loss=None` if you do not want your model to compute a loss. You can also specify `loss='auto'` to "
                "get the internal loss without printing this info string."
            )
            loss = "auto"
        if loss == "auto":
            loss = dummy_loss
            self._using_dummy_loss = True
        else:
            self._using_dummy_loss = False
        parent_args = list(inspect.signature(keras.Model.compile).parameters.keys())
        # This argument got renamed, we need to support both versions
        if "steps_per_execution" in parent_args:
            super().compile(
                optimizer=optimizer,
                loss=loss,
                metrics=metrics,
                loss_weights=loss_weights,
                weighted_metrics=weighted_metrics,
                run_eagerly=run_eagerly,
                steps_per_execution=steps_per_execution,
                **kwargs,
            )
        else:
            super().compile(
                optimizer=optimizer,
                loss=loss,
                metrics=metrics,
                loss_weights=loss_weights,
                weighted_metrics=weighted_metrics,
                run_eagerly=run_eagerly,
                experimental_steps_per_execution=steps_per_execution,
                **kwargs,
            )

    def compute_loss(self, *args, **kwargs):
        if hasattr(keras.Model, "compute_loss"):
            # This will be true in TF 2.8 or greater
            return super().compute_loss(*args, **kwargs)
        else:
            warnings.warn(
                "The old compute_loss method is deprecated as it conflicts with the Keras compute_loss "
                "method added in TF 2.8. If you want the original HF compute_loss, please call "
                "hf_compute_loss() instead. From TF versions >= 2.8, or Transformers versions >= 5, "
                "calling compute_loss() will get the Keras method instead.",
                FutureWarning,
            )
            return self.hf_compute_loss(*args, **kwargs)

    def get_label_to_output_name_mapping(self):
        arg_names = list(inspect.signature(self.call).parameters)
        if self._label_to_output_map is not None:
            return self._label_to_output_map
        elif "start_positions" in arg_names:
            return {"start_positions": "start_logits", "end_positions": "end_logits"}
        elif "sentence_order_label" in arg_names:
            return {"labels": "prediction_logits", "sentence_order_label": "sop_logits"}
        elif "next_sentence_label" in arg_names:
            return {"labels": "prediction_logits", "next_sentence_label": "seq_relationship_logits"}
        elif "mc_labels" in arg_names:
            return {"labels": "logits", "mc_labels": "mc_logits"}
        else:
            return {}

    def train_step(self, data):
        """
        A modification of Keras's default `train_step` that correctly handles matching outputs to labels for our models
        and supports directly training on the loss output head. In addition, it ensures input keys are copied to the
        labels where appropriate. It will also copy label keys into the input dict when using the dummy loss, to ensure
        that they are available to the model during the forward pass.
        """

        # We hardcode the most common renamings; models with weirder names can set `self._label_to_output_map`
        arg_names = list(inspect.signature(self.call).parameters)
        label_kwargs = find_labels(self.__class__)
        label_to_output = self.get_label_to_output_name_mapping()
        output_to_label = {val: key for key, val in label_to_output.items()}
        if not self._using_dummy_loss and parse(tf.__version__) < parse("2.11.0"):
            # Newer TF train steps leave this out
            data = expand_1d(data)
        x, y, sample_weight = keras.utils.unpack_x_y_sample_weight(data)
        # If the inputs are mutable dictionaries, make a shallow copy of them because we will modify
        # them during input/label pre-processing. This avoids surprising the user by wrecking their data.
        # In addition, modifying mutable Python inputs makes XLA compilation impossible.
        if isinstance(x, dict):
            x = x.copy()
        if isinstance(y, dict):
            y = y.copy()

        # When using a dummy loss, we ensure that separate labels are copied to the correct model arguments,
        # if those keys are not already present in the input dict
        if self._using_dummy_loss and y is not None:
            # If y is a tensor and the model only has one label-like input, map y to that input
            if len(label_kwargs) == 1 and isinstance(y, tf.Tensor):
                if isinstance(x, tf.Tensor):
                    x = {arg_names[0]: x}
                label_kwarg = next(iter(label_kwargs))
                if label_kwarg not in x:
                    x[label_kwarg] = y
            # Otherwise, copy keys from y to x as long as they weren't already present in x
            elif isinstance(y, dict):
                if isinstance(x, tf.Tensor):
                    x = {arg_names[0]: x}
                for key, val in y.items():
                    if key in arg_names and key not in x:
                        x[key] = val
                    elif output_to_label.get(key, None) in arg_names and key not in x:
                        x[output_to_label[key]] = val
        if y is None:
            y = {key: val for key, val in x.items() if key in label_kwargs}
            if not y and not self._using_dummy_loss:
                raise ValueError("Could not find label column(s) in input dict and no separate labels were provided!")

        if isinstance(y, dict):
            # Rename labels at this point to match output heads
            y = {label_to_output.get(key, key): val for key, val in y.items()}

        # Run forward pass.
        with tf.GradientTape() as tape:
            if self._using_dummy_loss and "return_loss" in arg_names:
                y_pred = self(x, training=True, return_loss=True)
            else:
                y_pred = self(x, training=True)
            if self._using_dummy_loss:
                loss = self.compiled_loss(y_pred.loss, y_pred.loss, sample_weight, regularization_losses=self.losses)
            else:
                loss = None

            # This next block matches outputs to label keys. Tensorflow's standard method for doing this
            # can get very confused if any of the keys contain nested values (e.g. lists/tuples of Tensors)
            if isinstance(y, dict) and len(y) == 1:
                if list(y.keys())[0] in y_pred.keys():
                    y_pred = y_pred[list(y.keys())[0]]
                elif list(y_pred.keys())[0] == "loss":
                    y_pred = y_pred[1]
                else:
                    y_pred = y_pred[0]
                _, y = y.popitem()
            elif isinstance(y, dict):
                # If the labels are a dict, match keys from the output by name
                y_pred = {key: val for key, val in y_pred.items() if key in y}
            elif isinstance(y, tuple) or isinstance(y, list):
                # If the labels are a tuple/list, match keys to the output by order, skipping the loss.
                if list(y_pred.keys())[0] == "loss":
                    y_pred = y_pred.to_tuple()[1:]
                else:
                    y_pred = y_pred.to_tuple()
                y_pred = y_pred[: len(y)]  # Remove unused fields in case those cause problems
            else:
                # If the labels are a single tensor, match them to the first non-loss tensor in the output
                if list(y_pred.keys())[0] == "loss":
                    y_pred = y_pred[1]
                else:
                    y_pred = y_pred[0]

            if loss is None:
                loss = self.compiled_loss(y, y_pred, sample_weight, regularization_losses=self.losses)

        # Run backwards pass.
        self.optimizer.minimize(loss, self.trainable_variables, tape=tape)

        self.compiled_metrics.update_state(y, y_pred, sample_weight)
        # Collect metrics to return
        return_metrics = {}
        for metric in self.metrics:
            result = metric.result()
            if isinstance(result, dict):
                return_metrics.update(result)
            else:
                return_metrics[metric.name] = result
        return return_metrics

    def test_step(self, data):
        """
        A modification of Keras's default `train_step` that correctly handles matching outputs to labels for our models
        and supports directly training on the loss output head. In addition, it ensures input keys are copied to the
        labels where appropriate. It will also copy label keys into the input dict when using the dummy loss, to ensure
        that they are available to the model during the forward pass.
        """
        # We hardcode the most common renamings; models with weirder names can set `self._label_to_output_map`
        arg_names = list(inspect.signature(self.call).parameters)
        label_kwargs = find_labels(self.__class__)
        label_to_output = self.get_label_to_output_name_mapping()
        output_to_label = {val: key for key, val in label_to_output.items()}
        if not self._using_dummy_loss and parse(tf.__version__) < parse("2.11.0"):
            # Newer versions leave this out
            data = expand_1d(data)
        x, y, sample_weight = keras.utils.unpack_x_y_sample_weight(data)
        # If the inputs are mutable dictionaries, make a shallow copy of them because we will modify
        # them during input/label pre-processing. This avoids surprising the user by wrecking their data.
        # In addition, modifying mutable Python inputs makes XLA compilation impossible.
        if isinstance(x, dict):
            x = x.copy()
        if isinstance(y, dict):
            y = y.copy()

        # When using a dummy loss, we ensure that separate labels are copied to the correct model arguments,
        # if those keys are not already present in the input dict
        if self._using_dummy_loss and y is not None:
            arg_names = list(inspect.signature(self.call).parameters)
            # If y is a tensor and the model only has one label-like input, map y to that input
            if len(label_kwargs) == 1 and isinstance(y, tf.Tensor):
                if isinstance(x, tf.Tensor):
                    x = {arg_names[0]: x}
                label_kwarg = next(iter(label_kwargs))
                if label_kwarg not in x:
                    x[label_kwarg] = y
            # Otherwise, copy keys from y to x as long as they weren't already present in x
            elif isinstance(y, dict):
                if isinstance(x, tf.Tensor):
                    x = {arg_names[0]: x}
                for key, val in y.items():
                    if key in arg_names and key not in x:
                        x[key] = val
                    elif output_to_label.get(key, None) in arg_names and key not in x:
                        x[output_to_label[key]] = val
        if y is None:
            y = {key: val for key, val in x.items() if key in label_kwargs}
            if not y and not self._using_dummy_loss:
                raise ValueError("Could not find label column(s) in input dict and no separate labels were provided!")

        if isinstance(y, dict):
            # Rename labels at this point to match output heads
            y = {label_to_output.get(key, key): val for key, val in y.items()}

        # Run forward pass.
        if self._using_dummy_loss and "return_loss" in arg_names:
            y_pred = self(x, return_loss=True, training=False)
        else:
            y_pred = self(x, training=False)
        if self._using_dummy_loss:
            loss = self.compiled_loss(y_pred.loss, y_pred.loss, sample_weight, regularization_losses=self.losses)
        else:
            loss = None

        # This next block matches outputs to label keys. Tensorflow's standard method for doing this
        # can get very confused if any of the keys contain nested values (e.g. lists/tuples of Tensors)
        if isinstance(y, dict) and len(y) == 1:
            if list(y.keys())[0] in y_pred.keys():
                y_pred = y_pred[list(y.keys())[0]]
            elif list(y_pred.keys())[0] == "loss":
                y_pred = y_pred[1]
            else:
                y_pred = y_pred[0]
            _, y = y.popitem()
        elif isinstance(y, dict):
            # If the labels are a dict, match keys from the output by name
            y_pred = {key: val for key, val in y_pred.items() if key in y}
        elif isinstance(y, tuple) or isinstance(y, list):
            # If the labels are a tuple/list, match keys to the output by order, skipping the loss.
            if list(y_pred.keys())[0] == "loss":
                y_pred = y_pred.to_tuple()[1:]
            else:
                y_pred = y_pred.to_tuple()
            y_pred = y_pred[: len(y)]  # Remove unused fields in case those cause problems
        else:
            # If the labels are a single tensor, match them to the first non-loss tensor in the output
            if list(y_pred.keys())[0] == "loss":
                y_pred = y_pred[1]
            else:
                y_pred = y_pred[0]

        if loss is None:
            loss = self.compiled_loss(y, y_pred, sample_weight, regularization_losses=self.losses)

        self.compiled_metrics.update_state(y, y_pred, sample_weight)
        # Collect metrics to return
        return_metrics = {}
        for metric in self.metrics:
            result = metric.result()
            if isinstance(result, dict):
                return_metrics.update(result)
            else:
                return_metrics[metric.name] = result
        return return_metrics

    def create_model_card(
        self,
        output_dir,
        model_name: str,
        language: Optional[str] = None,
        license: Optional[str] = None,
        tags: Optional[str] = None,
        finetuned_from: Optional[str] = None,
        tasks: Optional[str] = None,
        dataset_tags: Optional[Union[str, List[str]]] = None,
        dataset: Optional[Union[str, List[str]]] = None,
        dataset_args: Optional[Union[str, List[str]]] = None,
    ):
        """
        Creates a draft of a model card using the information available to the `Trainer`.

        Args:
            output_dir (`str` or `os.PathLike`):
                The folder in which to create the model card.
            model_name (`str`, *optional*):
                The name of the model.
            language (`str`, *optional*):
                The language of the model (if applicable)
            license (`str`, *optional*):
                The license of the model. Will default to the license of the pretrained model used, if the original
                model given to the `Trainer` comes from a repo on the Hub.
            tags (`str` or `List[str]`, *optional*):
                Some tags to be included in the metadata of the model card.
            finetuned_from (`str`, *optional*):
                The name of the model used to fine-tune this one (if applicable). Will default to the name of the repo
                of the original model given to the `Trainer` (if it comes from the Hub).
            tasks (`str` or `List[str]`, *optional*):
                One or several task identifiers, to be included in the metadata of the model card.
            dataset_tags (`str` or `List[str]`, *optional*):
                One or several dataset tags, to be included in the metadata of the model card.
            dataset (`str` or `List[str]`, *optional*):
                One or several dataset identifiers, to be included in the metadata of the model card.
            dataset_args (`str` or `List[str]`, *optional*):
               One or several dataset arguments, to be included in the metadata of the model card.
        """
        # Avoids a circular import by doing this when necessary.
        from .modelcard import TrainingSummary  # tests_ignore

        training_summary = TrainingSummary.from_keras(
            self,
            keras_history=self.history,
            language=language,
            license=license,
            tags=tags,
            model_name=model_name,
            finetuned_from=finetuned_from,
            tasks=tasks,
            dataset_tags=dataset_tags,
            dataset=dataset,
            dataset_args=dataset_args,
        )
        model_card = training_summary.to_model_card()
        with open(os.path.join(output_dir, "README.md"), "w") as f:
            f.write(model_card)

    def set_input_embeddings(self, value):
        """
        Set model's input embeddings

        Args:
            value (`tf.Variable`):
                The new weights mapping hidden states to vocabulary.
        """
        main_layer = getattr(self, self.base_model_prefix)

        if main_layer is None:
            raise NotImplementedError("The model does not implements the base_model_prefix attribute.")

        try:
            main_layer.set_input_embeddings(value)
        except AttributeError:
            logger.info("Building the model")
            self.build_in_name_scope()
            main_layer.set_input_embeddings(value)

    def get_output_embeddings(self) -> Union[None, keras.layers.Layer]:
        """
        Returns the model's output embeddings

        Returns:
            `tf.Variable`: The new weights mapping vocabulary to hidden states.
        """
        if self.get_lm_head() is not None:
            lm_head = self.get_lm_head()

            try:
                return lm_head.get_output_embeddings()
            except AttributeError:
                logger.info("Building the model")
                self.build_in_name_scope()

                return lm_head().get_output_embeddings()

        return None  # Overwrite for models with output embeddings

    def set_output_embeddings(self, value):
        """
        Set model's output embeddings

        Args:
            value (`tf.Variable`):
                The new weights mapping hidden states to vocabulary.
        """
        if self.get_lm_head() is not None:
            lm_head = self.get_lm_head()
            try:
                lm_head.set_output_embeddings(value)
            except AttributeError:
                logger.info("Building the model")
                self.build_in_name_scope()
                lm_head.set_output_embeddings(value)

    def get_output_layer_with_bias(self) -> Union[None, keras.layers.Layer]:
        """
        Get the layer that handles a bias attribute in case the model has an LM head with weights tied to the
        embeddings

        Return:
            `keras.layers.Layer`: The layer that handles the bias, None if not an LM model.
        """
        warnings.warn(
            "The method get_output_layer_with_bias is deprecated. Please use `get_lm_head` instead.", FutureWarning
        )
        return self.get_lm_head()

    def get_prefix_bias_name(self) -> Union[None, str]:
        """
        Get the concatenated _prefix name of the bias from the model name to the parent layer

        Return:
            `str`: The _prefix name of the bias.
        """
        warnings.warn("The method get_prefix_bias_name is deprecated. Please use `get_bias` instead.", FutureWarning)
        return None

    def get_bias(self) -> Union[None, Dict[str, tf.Variable]]:
        """
        Dict of bias attached to an LM head. The key represents the name of the bias attribute.

        Return:
            `tf.Variable`: The weights representing the bias, None if not an LM model.
        """
        if self.get_lm_head() is not None:
            lm_head = self.get_lm_head()
            try:
                return lm_head.get_bias()
            except AttributeError:
                self.build_in_name_scope()

                return lm_head.get_bias()
        return None

    def set_bias(self, value):
        """
        Set all the bias in the LM head.

        Args:
            value (`Dict[tf.Variable]`):
                All the new bias attached to an LM head.
        """
        if self.get_lm_head() is not None:
            lm_head = self.get_lm_head()
            try:
                lm_head.set_bias(value)
            except AttributeError:
                self.build_in_name_scope()
                lm_head.set_bias(value)

    def get_lm_head(self) -> keras.layers.Layer:
        """
        The LM Head layer. This method must be overwritten by all the models that have a lm head.

        Return:
            `keras.layers.Layer`: The LM head layer if the model has one, None if not.
        """
        return None

    def resize_token_embeddings(
        self, new_num_tokens: Optional[int] = None
    ) -> Union[keras.layers.Embedding, tf.Variable]:
        """
        Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`.

        Takes care of tying weights embeddings afterwards if the model class has a `tie_weights()` method.

        Arguments:
            new_num_tokens (`int`, *optional*):
                The number of new tokens in the embedding matrix. Increasing the size will add newly initialized
                vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just
                returns a pointer to the input tokens without doing anything.

        Return:
            `tf.Variable` or `keras.layers.Embedding`: Pointer to the input tokens of the model.
        """
        # TODO (joao): flagged for replacement (by `_v2_resized_token_embeddings`) due to embeddings refactor

        # Run the new code path if the model has a keras embeddings layer
        if isinstance(self.get_input_embeddings(), keras.layers.Embedding):
            return self._v2_resized_token_embeddings(new_num_tokens)

        if new_num_tokens is None or new_num_tokens == self.config.vocab_size:
            return self._get_word_embedding_weight(self.get_input_embeddings())

        model_embeds = self._resize_token_embeddings(new_num_tokens)

        # Update base model and current model config
        self.config.vocab_size = new_num_tokens

        return model_embeds

    def _v2_resized_token_embeddings(self, new_num_tokens: Optional[int] = None) -> keras.layers.Embedding:
        """
        Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`.

        Arguments:
            new_num_tokens (`int`, *optional*):
                The number of new tokens in the embedding matrix. Increasing the size will add newly initialized
                vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just
                returns a pointer to the input tokens without doing anything.

        Return:
            `keras.layers.Embedding`: Pointer to the input tokens of the model.
        """
        if new_num_tokens is None or new_num_tokens == self.config.vocab_size:
            return self.get_input_embeddings()

        model_embeds = self._v2_resize_token_embeddings(new_num_tokens)

        # Update base model and current model config
        self.config.vocab_size = new_num_tokens

        return model_embeds

    def _get_word_embedding_weight(model, embedding_layer):
        # TODO (joao): flagged for delection due to embeddings refactor

        # If the variable holds the weights themselves, return them
        if isinstance(embedding_layer, tf.Tensor):
            return embedding_layer
        # Otherwise, try to get them from the layer's attributes

        embeds = getattr(embedding_layer, "weight", None)
        if embeds is not None:
            return embeds

        embeds = getattr(embedding_layer, "decoder", None)
        if embeds is not None:
            return embeds

        # The reason why the attributes don't exist might be
        # because the model is not built, so retry getting
        # the argument after building the model
        model.build_in_name_scope()

        embeds = getattr(embedding_layer, "weight", None)
        if embeds is not None:
            return embeds

        embeds = getattr(embedding_layer, "decoder", None)
        if embeds is not None:
            return embeds

        return None

    def _resize_token_embeddings(self, new_num_tokens):
        # TODO (joao): flagged for replacement (by `_v2_resize_token_embeddings`) due to embeddings refactor
        old_embeddings = self._get_word_embedding_weight(self.get_input_embeddings())
        new_embeddings = self._get_resized_embeddings(old_embeddings, new_num_tokens)

        # if word embeddings are not tied, make sure that lm head bias is resized as well
        if self.get_bias() is not None:
            old_lm_head_bias = self.get_bias()
            new_lm_head_bias = self._get_resized_lm_head_bias(old_lm_head_bias, new_num_tokens)

            self.set_bias(new_lm_head_bias)

        # if word embeddings are not tied, make sure that lm head decoder is resized as well
        if self.get_output_embeddings() is not None:
            old_lm_head_decoder = self._get_word_embedding_weight(self.get_output_embeddings())
            new_lm_head_decoder = self._get_resized_lm_head_decoder(old_lm_head_decoder, new_num_tokens)

            self.set_output_embeddings(new_lm_head_decoder)

        self.set_input_embeddings(new_embeddings)

        return self.get_input_embeddings()

    def _v2_resize_token_embeddings(self, new_num_tokens):
        old_embeddings = self.get_input_embeddings()
        new_embeddings = self._v2_get_resized_embeddings(old_embeddings, new_num_tokens)
        self.set_input_embeddings(new_embeddings)

        # If word embeddings are not tied, make sure that lm head bias is resized as well
        if self.get_bias() is not None:
            old_lm_head_bias = self.get_bias()
            new_lm_head_bias = self._v2_get_resized_lm_head_bias(old_lm_head_bias, new_num_tokens)
            self.set_bias(new_lm_head_bias)

        # If word embeddings are not tied, make sure that lm head decoder is resized as well.
        tied_weights = self.get_input_embeddings() == self.get_output_embeddings()
        if self.get_output_embeddings() is not None and not tied_weights:
            old_lm_head_decoder = self._get_word_embedding_weight(self.get_output_embeddings())
            # TODO (joao): this one probably needs a v2 version with other models
            new_lm_head_decoder = self._get_resized_lm_head_decoder(old_lm_head_decoder, new_num_tokens)
            self.set_output_embeddings(new_lm_head_decoder)

        return self.get_input_embeddings()

    def _get_resized_lm_head_bias(self, old_lm_head_bias, new_num_tokens):
        """
        Build a resized bias from the old ones. Increasing the size will add newly initialized vectors at the end.
        Reducing the size will remove vectors from the end

        Args:
            old_lm_head_bias (`tf.Variable`):
                Old lm head bias to be resized.
            new_num_tokens (`int`, *optional*):
                New number of tokens in the linear matrix.

                Increasing the size will add newly initialized vectors at the end. Reducing the size will remove
                vectors from the end. If not provided or `None`, just returns None

        Return:
            `tf.Variable`: Pointer to the resized bias.
        """
        # TODO (joao): flagged for replacement (by `_v2_get_resized_lm_head_bias`) due to embeddings refactor
        new_lm_head_bias = {}

        for attr, weight in old_lm_head_bias.items():
            first_dim, old_num_tokens = (None, shape_list(weight)[0]) if tf.rank(weight) == 1 else shape_list(weight)
            size_diff = new_num_tokens - old_num_tokens
            final_shape = [new_num_tokens] if first_dim is None else [first_dim, new_num_tokens]

            # initialize new bias
            if tf.math.greater(size_diff, 0):
                padding_shape = [[0, size_diff]] if first_dim is None else [[0, 0], [0, size_diff]]
                current_bias = tf.pad(weight.value(), tf.convert_to_tensor(padding_shape), constant_values=-1)
                num_tokens_to_copy = min(old_num_tokens, new_num_tokens)
                mask_shape = [num_tokens_to_copy] if first_dim is None else [1, num_tokens_to_copy]
                bias_mask = tf.fill(tf.convert_to_tensor(mask_shape), True)
                bias_mask = tf.pad(bias_mask, tf.convert_to_tensor(padding_shape), constant_values=False)
            else:
                slice_from = [0] if first_dim is None else [0, 0]
                current_bias = tf.slice(
                    weight.value(), tf.convert_to_tensor(slice_from), tf.convert_to_tensor(final_shape)
                )
                bias_mask = tf.fill(tf.convert_to_tensor(final_shape), True)

            new_bias = self.add_weight(
                shape=final_shape,
                initializer="zeros",
                trainable=True,
                name=weight.name.split(":")[0],
            )
            init_bias = tf.where(bias_mask, current_bias, new_bias.value())

            new_bias.assign(init_bias)
            new_lm_head_bias[attr] = new_bias

        return new_lm_head_bias

    def _v2_get_resized_lm_head_bias(
        self, old_lm_head_bias: Dict[str, tf.Variable], new_num_tokens: int
    ) -> Dict[str, tf.Tensor]:
        """
        Build a resized bias from the old ones. Increasing the size will add newly initialized vectors at the end.
        Reducing the size will remove vectors from the end

        Args:
            old_lm_head_bias (`Dict[str, tf.Variable]`):
                Old lm head bias to be resized.
            new_num_tokens (`int`):
                New number of tokens in the linear matrix. Increasing the size will add newly initialized vectors at
                the end. Reducing the size will remove vectors from the end.

        Return:
            `tf.Tensor`: Values for the resized bias.
        """
        new_lm_head_bias = {}

        for attr, weight in old_lm_head_bias.items():
            # Determine the size difference (depending on the shape)
            first_dim, old_num_tokens = (None, shape_list(weight)[0]) if tf.rank(weight) == 1 else shape_list(weight)
            size_diff = new_num_tokens - old_num_tokens

            # Copy the old bias values to the new bias
            if old_num_tokens > new_num_tokens:
                new_bias = weight.value()[..., :new_num_tokens]
            else:
                padding_shape = [[0, size_diff]] if first_dim is None else [[0, 0], [0, size_diff]]
                new_bias = tf.pad(weight.value(), tf.convert_to_tensor(padding_shape))

            new_lm_head_bias[attr] = new_bias
        return new_lm_head_bias

    def _get_resized_lm_head_decoder(self, old_lm_head_decoder, new_num_tokens):
        """
        Build a resized decoder from the old ones. Increasing the size will add newly initialized vectors at the end.
        Reducing the size will remove vectors from the end

        Args:
            old_lm_head_decoder (`tf.Variable`):
                Old lm head decoder to be resized.
            new_num_tokens (`int`, *optional*):
                New number of tokens in the linear matrix.

                Increasing the size will add newly initialized vectors at the end. Reducing the size will remove
                vectors from the end. If not provided or `None`, just returns None

        Return:
            `tf.Variable`: Pointer to the resized decoder or None if the output embeddings are different from the input
            ones.
        """
        new_lm_head_decoder = old_lm_head_decoder
        is_input_output_equals = tf.reduce_any(
            self._get_word_embedding_weight(self.get_input_embeddings()) == old_lm_head_decoder
        )

        if old_lm_head_decoder is not None and not is_input_output_equals:
            old_embedding_dim = shape_list(old_lm_head_decoder)[1]
            decoder_mask, current_decoder = init_copy_embeddings(old_lm_head_decoder, new_num_tokens)
            new_lm_head_decoder = self.add_weight(
                shape=(new_num_tokens, old_embedding_dim),
                initializer="zeros",
                trainable=True,
                name=old_lm_head_decoder.name.split(":")[0],
            )
            init_decoder = tf.where(decoder_mask, current_decoder, new_lm_head_decoder.value())

            new_lm_head_decoder.assign(init_decoder)

        return new_lm_head_decoder

    def _get_resized_embeddings(self, old_embeddings, new_num_tokens=None) -> tf.Variable:
        """
        Build a resized Embedding weights from a provided token Embedding weights. Increasing the size will add newly
        initialized vectors at the end. Reducing the size will remove vectors from the end

        Args:
            old_embeddings (`tf.Variable`):
                Old embeddings to be resized.
            new_num_tokens (`int`, *optional*):
                New number of tokens in the embedding matrix.

                Increasing the size will add newly initialized vectors at the end. Reducing the size will remove
                vectors from the end. If not provided or `None`, just returns a pointer to the input tokens
                `tf.Variable` module of the model without doing anything.

        Return:
            `tf.Variable`: Pointer to the resized Embedding Module or the old Embedding Module if `new_num_tokens` is
            `None`
        """
        # TODO (joao): flagged for replacement (by `_v2_get_resized_embeddings`) due to embeddings refactor
        old_embedding_dim = shape_list(old_embeddings)[1]
        init_range = getattr(self.config, "initializer_range", 0.02)
        embeddings_mask, current_embeddings = init_copy_embeddings(old_embeddings, new_num_tokens)
        new_embeddings = self.add_weight(
            name=old_embeddings.name.split(":")[0],
            shape=[new_num_tokens, old_embedding_dim],
            initializer=get_initializer(init_range),
            dtype=tf.float32,
        )
        init_embeddings = tf.where(embeddings_mask, current_embeddings, new_embeddings.value())

        new_embeddings.assign(init_embeddings)

        return new_embeddings

    def _v2_get_resized_embeddings(
        self, old_embeddings: keras.layers.Embedding, new_num_tokens: int
    ) -> keras.layers.Embedding:
        """
        Build a resized Embedding layer from a provided Embedding layer. Increasing the size will add newly initialized
        vectors at the end. Reducing the size will remove vectors from the end.

        Args:
            old_embeddings (`keras.layers.Embedding`):
                Old embeddings to be resized.
            new_num_tokens (`int`, *optional*):
                New number of tokens in the embedding matrix.

        Return:
            `keras.layers.Embedding`: Resized Embedding layer.
        """

        # Get the initialization range for the embeddings
        init_range = 0.02  # default value
        potential_initialization_variable_names = [
            "initializer_range",  # most common
            "initializer_factor",  # e.g. T5
            "init_std",  # e.g BART
        ]
        for var_name in potential_initialization_variable_names:
            if hasattr(self.config, var_name):
                init_range = getattr(self.config, var_name)

        # Get a new (initialized) embeddings layer
        new_embeddings = keras.layers.Embedding(
            input_dim=new_num_tokens,
            output_dim=old_embeddings.output_dim,
            embeddings_initializer=keras.initializers.TruncatedNormal(stddev=init_range),
            name=old_embeddings.embeddings.name[:-13],  # exact same scoped name except "/embeddings:0"
        )
        new_embeddings(tf.constant([[0]]))

        # Copy the old embeddings to the new embeddings
        if old_embeddings.input_dim >= new_num_tokens:
            init_embeddings = old_embeddings.embeddings[:new_num_tokens]
        else:
            init_embeddings = tf.concat(
                [old_embeddings.embeddings, new_embeddings.embeddings[old_embeddings.input_dim :]], axis=0
            )
        new_embeddings.embeddings.assign(init_embeddings)
        return new_embeddings

    def prune_heads(self, heads_to_prune):
        """
        Prunes heads of the base model.

        Arguments:
            heads_to_prune (`Dict[int, List[int]]`):
                Dictionary with keys being selected layer indices (`int`) and associated values being the list of heads
                to prune in said layer (list of `int`). For instance {1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on
                layer 1 and heads 2 and 3 on layer 2.
        """
        raise NotImplementedError

    def save_pretrained(
        self,
        save_directory,
        saved_model=False,
        version=1,
        push_to_hub=False,
        signatures=None,
        max_shard_size: Union[int, str] = "5GB",
        create_pr: bool = False,
        safe_serialization: bool = False,
        token: Optional[Union[str, bool]] = None,
        **kwargs,
    ):
        """
        Save a model and its configuration file to a directory, so that it can be re-loaded using the
        [`~TFPreTrainedModel.from_pretrained`] class method.

        Arguments:
            save_directory (`str`):
                Directory to which to save. Will be created if it doesn't exist.
            saved_model (`bool`, *optional*, defaults to `False`):
                If the model has to be saved in saved model format as well or not.
            version (`int`, *optional*, defaults to 1):
                The version of the saved model. A saved model needs to be versioned in order to be properly loaded by
                TensorFlow Serving as detailed in the official documentation
                https://www.tensorflow.org/tfx/serving/serving_basic
            push_to_hub (`bool`, *optional*, defaults to `False`):
                Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
                repository you want to push to with `repo_id` (will default to the name of `save_directory` in your
                namespace).
            signatures (`dict` or `tf.function`, *optional*):
                Model's signature used for serving. This will be passed to the `signatures` argument of model.save().
            max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`):
                The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size
                lower than this size. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`).

                <Tip warning={true}>

                If a single weight of the model is bigger than `max_shard_size`, it will be in its own checkpoint shard
                which will be bigger than `max_shard_size`.

                </Tip>

            create_pr (`bool`, *optional*, defaults to `False`):
                Whether or not to create a PR with the uploaded files or directly commit.
            safe_serialization (`bool`, *optional*, defaults to `False`):
                Whether to save the model using `safetensors` or the traditional TensorFlow way (that uses `h5`).
            token (`str` or `bool`, *optional*):
                The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use
                the token generated when running `huggingface-cli login` (stored in `~/.huggingface`).
            kwargs (`Dict[str, Any]`, *optional*):
                Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.
        """
        use_auth_token = kwargs.pop("use_auth_token", None)

        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token

        if token is not None:
            kwargs["token"] = token

        if os.path.isfile(save_directory):
            logger.error(f"Provided path ({save_directory}) should be a directory, not a file")
            return

        os.makedirs(save_directory, exist_ok=True)

        if push_to_hub:
            commit_message = kwargs.pop("commit_message", None)
            repo_id = kwargs.pop("repo_id", save_directory.split(os.path.sep)[-1])
            repo_id = self._create_repo(repo_id, **kwargs)
            files_timestamps = self._get_files_timestamps(save_directory)

        if saved_model:
            # If `torch_dtype` is in the config with a torch dtype class as the value, we need to change it to string.
            # (Although TF doesn't care about this attribute, we can't just remove it or set it to `None`.)
            if getattr(self.config, "torch_dtype", None) is not None and not isinstance(self.config.torch_dtype, str):
                self.config.torch_dtype = str(self.config.torch_dtype).split(".")[1]
            if signatures is None:
                serving_default = self.serving.get_concrete_function(self.input_signature)
                if any(spec.dtype == tf.int32 for spec in self.input_signature.values()):
                    int64_spec = {
                        key: tf.TensorSpec(
                            shape=spec.shape, dtype=tf.int64 if spec.dtype == tf.int32 else spec.dtype, name=spec.name
                        )
                        for key, spec in self.input_signature.items()
                    }
                    int64_serving = self.serving.get_concrete_function(int64_spec)
                    signatures = {"serving_default": serving_default, "int64_serving": int64_serving}
                else:
                    signatures = serving_default
            saved_model_dir = os.path.join(save_directory, "saved_model", str(version))
            self.save(saved_model_dir, include_optimizer=False, signatures=signatures)
            logger.info(f"Saved model created in {saved_model_dir}")

        # Save configuration file
        self.config.architectures = [self.__class__.__name__[2:]]

        # If we have a custom model, we copy the file defining it in the folder and set the attributes so it can be
        # loaded from the Hub.
        if self._auto_class is not None:
            custom_object_save(self, save_directory, config=self.config)

        self.config.save_pretrained(save_directory)
        if self.can_generate():
            self.generation_config.save_pretrained(save_directory)

        # If we save using the predefined names, we can load using `from_pretrained`
        weights_name = SAFE_WEIGHTS_NAME if safe_serialization else TF2_WEIGHTS_NAME
        output_model_file = os.path.join(save_directory, weights_name)

        shards, index = tf_shard_checkpoint(self.weights, max_shard_size, weights_name=weights_name)

        # Clean the folder from a previous save
        for filename in os.listdir(save_directory):
            full_filename = os.path.join(save_directory, filename)
            # If we have a shard file that is not going to be replaced, we delete it, but only from the main process
            # in distributed settings to avoid race conditions.
            weights_no_suffix = weights_name.replace(".bin", "").replace(".safetensors", "")
            if (
                filename.startswith(weights_no_suffix)
                and os.path.isfile(full_filename)
                and filename not in shards.keys()
            ):
                os.remove(full_filename)

        if index is None:
            if safe_serialization:
                state_dict = {strip_model_name_and_prefix(w.name): w.value() for w in self.weights}
                safe_save_file(state_dict, output_model_file, metadata={"format": "tf"})
            else:
                self.save_weights(output_model_file)
            logger.info(f"Model weights saved in {output_model_file}")
        else:
            save_index_file = SAFE_WEIGHTS_INDEX_NAME if safe_serialization else TF2_WEIGHTS_INDEX_NAME
            save_index_file = os.path.join(save_directory, save_index_file)
            # Save the index as well
            with open(save_index_file, "w", encoding="utf-8") as index_file:
                content = json.dumps(index, indent=2, sort_keys=True) + "\n"
                index_file.write(content)
            logger.info(
                f"The model is bigger than the maximum size per checkpoint ({max_shard_size}) and is going to be "
                f"split in {len(shards)} checkpoint shards. You can find where each parameters has been saved in the "
                f"index located at {save_index_file}."
            )
            for shard_file, shard in shards.items():
                if safe_serialization:
                    shard_state_dict = {strip_model_name_and_prefix(w.name): w.value() for w in shard}
                    safe_save_file(
                        shard_state_dict, os.path.join(save_directory, shard_file), metadata={"format": "tf"}
                    )
                else:
                    with h5py.File(os.path.join(save_directory, shard_file), mode="w") as shard_file:
                        layers = []
                        for layer in sorted(shard, key=lambda x: x.name):
                            if "model." in layer.name or len(layer.name.split("/")) == 1:
                                layer_name = layer.name
                            else:
                                layer_name = "/".join(layer.name.split("/")[1:])
                            param_dset = shard_file.create_dataset(
                                layer_name, layer.numpy().shape, dtype=layer.numpy().dtype
                            )
                            param_dset[:] = layer.numpy()
                            layers.append(layer_name.encode("utf8"))
                        save_attributes_to_hdf5_group(shard_file, "layer_names", layers)

        if push_to_hub:
            self._upload_modified_files(
                save_directory,
                repo_id,
                files_timestamps,
                commit_message=commit_message,
                token=token,
            )

    @classmethod
    def from_pretrained(
        cls,
        pretrained_model_name_or_path: Optional[Union[str, os.PathLike]],
        *model_args,
        config: Optional[Union[PretrainedConfig, str, os.PathLike]] = None,
        cache_dir: Optional[Union[str, os.PathLike]] = None,
        ignore_mismatched_sizes: bool = False,
        force_download: bool = False,
        local_files_only: bool = False,
        token: Optional[Union[str, bool]] = None,
        revision: str = "main",
        use_safetensors: bool = None,
        **kwargs,
    ):
        r"""
        Instantiate a pretrained TF 2.0 model from a pre-trained model configuration.

        The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come
        pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
        task.

        The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those
        weights are discarded.

        Parameters:
            pretrained_model_name_or_path (`str`, *optional*):
                Can be either:

                    - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
                    - A path to a *directory* containing model weights saved using
                      [`~TFPreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`.
                    - A path or url to a *PyTorch state_dict save file* (e.g, `./pt_model/pytorch_model.bin`). In this
                      case, `from_pt` should be set to `True` and a configuration object should be provided as `config`
                      argument. This loading path is slower than converting the PyTorch model in a TensorFlow model
                      using the provided conversion scripts and loading the TensorFlow model afterwards.
                    - `None` if you are both providing the configuration and state dictionary (resp. with keyword
                      arguments `config` and `state_dict`).
            model_args (sequence of positional arguments, *optional*):
                All remaining positional arguments will be passed to the underlying model's `__init__` method.
            config (`Union[PretrainedConfig, str]`, *optional*):
                Can be either:

                    - an instance of a class derived from [`PretrainedConfig`],
                    - a string valid as input to [`~PretrainedConfig.from_pretrained`].

                Configuration for the model to use instead of an automatically loaded configuration. Configuration can
                be automatically loaded when:

                    - The model is a model provided by the library (loaded with the *model id* string of a pretrained
                      model).
                    - The model was saved using [`~TFPreTrainedModel.save_pretrained`] and is reloaded by supplying the
                      save directory.
                    - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a
                      configuration JSON file named *config.json* is found in the directory.
            from_pt (`bool`, *optional*, defaults to `False`):
                Load the model weights from a PyTorch state_dict save file (see docstring of
                `pretrained_model_name_or_path` argument).
            ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`):
                Whether or not to raise an error if some of the weights from the checkpoint do not have the same size
                as the weights of the model (if for instance, you are instantiating a model with 10 labels from a
                checkpoint with 3 labels).
            cache_dir (`str`, *optional*):
                Path to a directory in which a downloaded pretrained model configuration should be cached if the
                standard cache should not be used.
            force_download (`bool`, *optional*, defaults to `False`):
                Whether or not to force the (re-)download of the model weights and configuration files, overriding the
                cached versions if they exist.
            resume_download:
                Deprecated and ignored. All downloads are now resumed by default when possible.
                Will be removed in v5 of Transformers.
            proxies:
                (`Dict[str, str], `optional`): A dictionary of proxy servers to use by protocol or endpoint, e.g.,
                `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
                output_loading_info(`bool`, *optional*, defaults to `False`): Whether ot not to also return a
                dictionary containing missing keys, unexpected keys and error messages.
            local_files_only(`bool`, *optional*, defaults to `False`):
                Whether or not to only look at local files (e.g., not try downloading the model).
            token (`str` or `bool`, *optional*):
                The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use
                the token generated when running `huggingface-cli login` (stored in `~/.huggingface`).
            revision (`str`, *optional*, defaults to `"main"`):
                The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
                git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
                identifier allowed by git.


                <Tip>

                To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>"`.

                </Tip>

            mirror (`str`, *optional*):
                Mirror source to accelerate downloads in China. If you are from China and have an accessibility
                problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety.
                Please refer to the mirror site for more information.
            subfolder (`str`, *optional*, defaults to `""`):
                In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
                specify the folder name here.
            tf_to_pt_weight_rename (`Callable`, *optional*):
                A function that is called to transform the names of weights during the PyTorch to TensorFlow
                crossloading process. This is not necessary for most models, but is useful to allow composite models to
                be crossloaded correctly.
            use_safetensors (`bool`, *optional*, defaults to `None`):
                Whether or not to use `safetensors` checkpoints. Defaults to `None`. If not specified and `safetensors`
                is not installed, it will be set to `False`.
            kwargs (remaining dictionary of keyword arguments, *optional*):
                Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
                `output_attentions=True`). Behaves differently depending on whether a `config` is provided or
                automatically loaded:

                    - If a configuration is provided with `config`, `**kwargs` will be directly passed to the
                      underlying model's `__init__` method (we assume all relevant updates to the configuration have
                      already been done)
                    - If a configuration is not provided, `kwargs` will be first passed to the configuration class
                      initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that
                      corresponds to a configuration attribute will be used to override said attribute with the
                      supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute
                      will be passed to the underlying model's `__init__` function.

        Examples:

        ```python
        >>> from transformers import BertConfig, TFBertModel

        >>> # Download model and configuration from huggingface.co and cache.
        >>> model = TFBertModel.from_pretrained("google-bert/bert-base-uncased")
        >>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
        >>> model = TFBertModel.from_pretrained("./test/saved_model/")
        >>> # Update configuration during loading.
        >>> model = TFBertModel.from_pretrained("google-bert/bert-base-uncased", output_attentions=True)
        >>> assert model.config.output_attentions == True
        >>> # Loading from a Pytorch model file instead of a TensorFlow checkpoint (slower, for example purposes, not runnable).
        >>> config = BertConfig.from_json_file("./pt_model/my_pt_model_config.json")
        >>> model = TFBertModel.from_pretrained("./pt_model/my_pytorch_model.bin", from_pt=True, config=config)
        ```"""
        from_pt = kwargs.pop("from_pt", False)
        resume_download = kwargs.pop("resume_download", None)
        proxies = kwargs.pop("proxies", None)
        output_loading_info = kwargs.pop("output_loading_info", False)
        use_auth_token = kwargs.pop("use_auth_token", None)
        trust_remote_code = kwargs.pop("trust_remote_code", None)
        _ = kwargs.pop("mirror", None)
        load_weight_prefix = kwargs.pop("load_weight_prefix", None)
        from_pipeline = kwargs.pop("_from_pipeline", None)
        from_auto_class = kwargs.pop("_from_auto", False)
        subfolder = kwargs.pop("subfolder", "")
        commit_hash = kwargs.pop("_commit_hash", None)
        tf_to_pt_weight_rename = kwargs.pop("tf_to_pt_weight_rename", None)

        # Not relevant for TF models
        _ = kwargs.pop("adapter_kwargs", None)

        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token

        if trust_remote_code is True:
            logger.warning(
                "The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is"
                " ignored."
            )

        user_agent = {"file_type": "model", "framework": "tensorflow", "from_auto_class": from_auto_class}
        if from_pipeline is not None:
            user_agent["using_pipeline"] = from_pipeline

        if is_offline_mode() and not local_files_only:
            logger.info("Offline mode: forcing local_files_only=True")
            local_files_only = True

        if use_safetensors is None and not is_safetensors_available():
            use_safetensors = False

        # Load config if we don't provide a configuration
        if not isinstance(config, PretrainedConfig):
            config_path = config if config is not None else pretrained_model_name_or_path
            config, model_kwargs = cls.config_class.from_pretrained(
                config_path,
                cache_dir=cache_dir,
                return_unused_kwargs=True,
                force_download=force_download,
                resume_download=resume_download,
                proxies=proxies,
                local_files_only=local_files_only,
                token=token,
                revision=revision,
                _from_auto=from_auto_class,
                _from_pipeline=from_pipeline,
                _commit_hash=commit_hash,
                **kwargs,
            )
        else:
            model_kwargs = kwargs

        if commit_hash is None:
            commit_hash = getattr(config, "_commit_hash", None)

        # This variable will flag if we're loading a sharded checkpoint. In this case the archive file is just the
        # index of the files.
        is_sharded = False
        # Load model
        if pretrained_model_name_or_path is not None:
            pretrained_model_name_or_path = str(pretrained_model_name_or_path)
            is_local = os.path.isdir(pretrained_model_name_or_path)
            if is_local:
                if from_pt and os.path.isfile(os.path.join(pretrained_model_name_or_path, WEIGHTS_NAME)):
                    # Load from a PyTorch checkpoint in priority if from_pt
                    archive_file = os.path.join(pretrained_model_name_or_path, WEIGHTS_NAME)
                elif from_pt and os.path.isfile(os.path.join(pretrained_model_name_or_path, WEIGHTS_INDEX_NAME)):
                    # Load from a sharded PyTorch checkpoint
                    archive_file = os.path.join(pretrained_model_name_or_path, WEIGHTS_INDEX_NAME)
                    is_sharded = True
                elif use_safetensors is not False and os.path.isfile(
                    os.path.join(pretrained_model_name_or_path, SAFE_WEIGHTS_NAME)
                ):
                    # Load from a safetensors checkpoint
                    archive_file = os.path.join(pretrained_model_name_or_path, SAFE_WEIGHTS_NAME)
                elif use_safetensors is not False and os.path.isfile(
                    os.path.join(pretrained_model_name_or_path, SAFE_WEIGHTS_INDEX_NAME)
                ):
                    # Load from a sharded safetensors checkpoint
                    archive_file = os.path.join(pretrained_model_name_or_path, SAFE_WEIGHTS_INDEX_NAME)
                    is_sharded = True
                elif os.path.isfile(os.path.join(pretrained_model_name_or_path, TF2_WEIGHTS_NAME)):
                    # Load from a TF 2.0 checkpoint
                    archive_file = os.path.join(pretrained_model_name_or_path, TF2_WEIGHTS_NAME)
                elif os.path.isfile(os.path.join(pretrained_model_name_or_path, TF2_WEIGHTS_INDEX_NAME)):
                    # Load from a sharded TF 2.0 checkpoint
                    archive_file = os.path.join(pretrained_model_name_or_path, TF2_WEIGHTS_INDEX_NAME)
                    is_sharded = True

                # At this stage we don't have a weight file so we will raise an error.
                elif use_safetensors:
                    raise EnvironmentError(
                        f"Error no file named {SAFE_WEIGHTS_NAME} or {SAFE_WEIGHTS_INDEX_NAME} found in directory {pretrained_model_name_or_path}. "
                        f"Please make sure that the model has been saved with `safe_serialization=True` or do not "
                        f"set `use_safetensors=True`."
                    )
                elif os.path.isfile(os.path.join(pretrained_model_name_or_path, WEIGHTS_NAME)) or os.path.isfile(
                    os.path.join(pretrained_model_name_or_path, WEIGHTS_INDEX_NAME)
                ):
                    raise EnvironmentError(
                        f"Error no file named {TF2_WEIGHTS_NAME} or {SAFE_WEIGHTS_NAME} found in directory {pretrained_model_name_or_path} "
                        "but there is a file for PyTorch weights. Use `from_pt=True` to load this model from those "
                        "weights."
                    )
                else:
                    raise EnvironmentError(
                        f"Error no file named {TF2_WEIGHTS_NAME}, {SAFE_WEIGHTS_NAME} or {WEIGHTS_NAME} found in directory "
                        f"{pretrained_model_name_or_path}."
                    )
            elif os.path.isfile(pretrained_model_name_or_path):
                archive_file = pretrained_model_name_or_path
                is_local = True
            elif os.path.isfile(pretrained_model_name_or_path + ".index"):
                archive_file = pretrained_model_name_or_path + ".index"
                is_local = True
            elif is_remote_url(pretrained_model_name_or_path):
                filename = pretrained_model_name_or_path
                resolved_archive_file = download_url(pretrained_model_name_or_path)
            else:
                # set correct filename
                if from_pt:
                    filename = WEIGHTS_NAME
                elif use_safetensors is not False:
                    filename = SAFE_WEIGHTS_NAME
                else:
                    filename = TF2_WEIGHTS_NAME

                try:
                    # Load from URL or cache if already cached
                    cached_file_kwargs = {
                        "cache_dir": cache_dir,
                        "force_download": force_download,
                        "proxies": proxies,
                        "resume_download": resume_download,
                        "local_files_only": local_files_only,
                        "token": token,
                        "user_agent": user_agent,
                        "revision": revision,
                        "subfolder": subfolder,
                        "_raise_exceptions_for_gated_repo": False,
                        "_raise_exceptions_for_missing_entries": False,
                        "_commit_hash": commit_hash,
                    }
                    resolved_archive_file = cached_file(pretrained_model_name_or_path, filename, **cached_file_kwargs)

                    # Since we set _raise_exceptions_for_missing_entries=False, we don't get an exception but a None
                    # result when internet is up, the repo and revision exist, but the file does not.
                    if resolved_archive_file is None and filename == SAFE_WEIGHTS_NAME:
                        # Did not find the safetensors file, let's fallback to TF.
                        # No support for sharded safetensors yet, so we'll raise an error if that's all we find.
                        filename = TF2_WEIGHTS_NAME
                        resolved_archive_file = cached_file(
                            pretrained_model_name_or_path, TF2_WEIGHTS_NAME, **cached_file_kwargs
                        )
                    if resolved_archive_file is None and filename == TF2_WEIGHTS_NAME:
                        # Maybe the checkpoint is sharded, we try to grab the index name in this case.
                        resolved_archive_file = cached_file(
                            pretrained_model_name_or_path, TF2_WEIGHTS_INDEX_NAME, **cached_file_kwargs
                        )
                        if resolved_archive_file is not None:
                            is_sharded = True
                    if resolved_archive_file is None and filename == WEIGHTS_NAME:
                        # Maybe the checkpoint is sharded, we try to grab the index name in this case.
                        resolved_archive_file = cached_file(
                            pretrained_model_name_or_path, WEIGHTS_INDEX_NAME, **cached_file_kwargs
                        )
                        if resolved_archive_file is not None:
                            is_sharded = True
                    if resolved_archive_file is None:
                        # Otherwise, maybe there is a PyTorch or Flax model file.  We try those to give a helpful error
                        # message.
                        has_file_kwargs = {
                            "revision": revision,
                            "proxies": proxies,
                            "token": token,
                            "cache_dir": cache_dir,
                            "local_files_only": local_files_only,
                        }
                        if has_file(pretrained_model_name_or_path, SAFE_WEIGHTS_INDEX_NAME, **has_file_kwargs):
                            is_sharded = True
                        elif has_file(pretrained_model_name_or_path, WEIGHTS_NAME, **has_file_kwargs):
                            raise EnvironmentError(
                                f"{pretrained_model_name_or_path} does not appear to have a file named"
                                f" {TF2_WEIGHTS_NAME} but there is a file for PyTorch weights. Use `from_pt=True` to"
                                " load this model from those weights."
                            )
                        else:
                            raise EnvironmentError(
                                f"{pretrained_model_name_or_path} does not appear to have a file named {WEIGHTS_NAME},"
                                f" {TF2_WEIGHTS_NAME} or {TF_WEIGHTS_NAME}"
                            )

                except EnvironmentError:
                    # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted
                    # to the original exception.
                    raise
                except Exception:
                    # For any other exception, we throw a generic error.

                    raise EnvironmentError(
                        f"Can't load the model for '{pretrained_model_name_or_path}'. If you were trying to load it"
                        " from 'https://huggingface.co/models', make sure you don't have a local directory with the"
                        f" same name. Otherwise, make sure '{pretrained_model_name_or_path}' is the correct path to a"
                        f" directory containing a file named {WEIGHTS_NAME}, {TF2_WEIGHTS_NAME} or {TF_WEIGHTS_NAME}"
                    )
            if is_local:
                logger.info(f"loading weights file {archive_file}")
                resolved_archive_file = archive_file
                filename = resolved_archive_file.split(os.path.sep)[-1]
            else:
                logger.info(f"loading weights file {filename} from cache at {resolved_archive_file}")
        else:
            resolved_archive_file = None

        # We'll need to download and cache each checkpoint shard if the checkpoint is sharded.
        if is_sharded:
            # resolved_archive_file becomes a list of files that point to the different checkpoint shards in this case.
            resolved_archive_file, sharded_metadata = get_checkpoint_shard_files(
                pretrained_model_name_or_path,
                resolved_archive_file,
                cache_dir=cache_dir,
                force_download=force_download,
                proxies=proxies,
                resume_download=resume_download,
                local_files_only=local_files_only,
                token=token,
                user_agent=user_agent,
                revision=revision,
                _commit_hash=commit_hash,
            )

        safetensors_from_pt = False
        if filename == SAFE_WEIGHTS_NAME:
            with safe_open(resolved_archive_file, framework="tf") as f:
                safetensors_metadata = f.metadata()
            if safetensors_metadata is None or safetensors_metadata.get("format") not in ["pt", "tf", "flax", "mlx"]:
                raise OSError(
                    f"The safetensors archive passed at {resolved_archive_file} does not contain the valid metadata."
                    " Make sure you save your model with the `save_pretrained` method."
                )
            safetensors_from_pt = safetensors_metadata.get("format") == "pt"
        elif filename == SAFE_WEIGHTS_INDEX_NAME:
            with safe_open(resolved_archive_file[0], framework="tf") as f:
                safetensors_metadata = f.metadata()
            if safetensors_metadata is None or safetensors_metadata.get("format") not in ["pt", "tf", "flax", "mlx"]:
                raise OSError(
                    f"The safetensors archive passed at {resolved_archive_file} does not contain the valid metadata."
                    " Make sure you save your model with the `save_pretrained` method."
                )
            safetensors_from_pt = safetensors_metadata.get("format") == "pt"

        config.name_or_path = pretrained_model_name_or_path

        # composed models, *e.g.* TFRag, require special treatment when it comes to loading
        # pre-trained weights.
        if cls._requires_load_weight_prefix and model_kwargs.get("name") is not None:
            model_kwargs["load_weight_prefix"] = load_weight_prefix + "/" + model_kwargs.get("name")

        # Instantiate model.
        model = cls(config, *model_args, **model_kwargs)

        if tf_to_pt_weight_rename is None and hasattr(model, "tf_to_pt_weight_rename"):
            # TODO Matt: This is a temporary workaround to allow weight renaming, but requires a method
            #            to be defined for each class that requires a rename. We can probably just have a class-level
            #            dict and a single top-level method or something and cut down a lot of boilerplate code
            tf_to_pt_weight_rename = model.tf_to_pt_weight_rename

        if from_pt:
            from .modeling_tf_pytorch_utils import load_pytorch_checkpoint_in_tf2_model

            # Load from a PyTorch checkpoint
            return load_pytorch_checkpoint_in_tf2_model(
                model,
                resolved_archive_file,
                allow_missing_keys=True,
                output_loading_info=output_loading_info,
                _prefix=load_weight_prefix,
                tf_to_pt_weight_rename=tf_to_pt_weight_rename,
            )

        # we might need to extend the variable scope for composite models
        if load_weight_prefix is not None:
            with tf.compat.v1.variable_scope(load_weight_prefix):
                model.build_in_name_scope()  # build the network with dummy inputs
        else:
            model.build_in_name_scope()  # build the network with dummy inputs

        if safetensors_from_pt and not is_sharded:
            from .modeling_tf_pytorch_utils import load_pytorch_state_dict_in_tf2_model

            with safe_open(resolved_archive_file, framework="tf") as safetensors_archive:
                # Load from a PyTorch safetensors checkpoint
                # We load in TF format here because PT weights often need to be transposed, and this is much
                # faster on GPU. Loading as numpy and transposing on CPU adds several seconds to load times.
                return load_pytorch_state_dict_in_tf2_model(
                    model,
                    safetensors_archive,
                    tf_inputs=False,  # No need to build the model again
                    allow_missing_keys=True,
                    output_loading_info=output_loading_info,
                    _prefix=load_weight_prefix,
                    ignore_mismatched_sizes=ignore_mismatched_sizes,
                    tf_to_pt_weight_rename=tf_to_pt_weight_rename,
                )
        elif safetensors_from_pt:
            from .modeling_tf_pytorch_utils import load_sharded_pytorch_safetensors_in_tf2_model

            return load_sharded_pytorch_safetensors_in_tf2_model(
                model,
                resolved_archive_file,
                tf_inputs=False,
                allow_missing_keys=True,
                output_loading_info=output_loading_info,
                _prefix=load_weight_prefix,
                ignore_mismatched_sizes=ignore_mismatched_sizes,
                tf_to_pt_weight_rename=tf_to_pt_weight_rename,
            )

        # 'by_name' allow us to do transfer learning by skipping/adding layers
        # see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1339-L1357
        try:
            if is_sharded:
                for file in resolved_archive_file:
                    os.path.isfile(file), f"Error retrieving files {file}"
                if filename == SAFE_WEIGHTS_INDEX_NAME:
                    missing_keys, unexpected_keys, mismatched_keys = load_tf_sharded_weights_from_safetensors(
                        model,
                        resolved_archive_file,
                        ignore_mismatched_sizes=ignore_mismatched_sizes,
                        _prefix=load_weight_prefix,
                    )
                else:
                    missing_keys, unexpected_keys, mismatched_keys = load_tf_sharded_weights(
                        model,
                        resolved_archive_file,
                        ignore_mismatched_sizes=ignore_mismatched_sizes,
                        _prefix=load_weight_prefix,
                    )
            else:
                # Handles both H5 and safetensors
                missing_keys, unexpected_keys, mismatched_keys = load_tf_weights(
                    model,
                    resolved_archive_file,
                    ignore_mismatched_sizes=ignore_mismatched_sizes,
                    _prefix=load_weight_prefix,
                )
        except OSError as e:
            try:
                with open(resolved_archive_file) as f:
                    if f.read().startswith("version"):
                        raise OSError(
                            "You seem to have cloned a repository without having git-lfs installed. Please install "
                            "git-lfs and run `git lfs install` followed by `git lfs pull` in the folder "
                            "you cloned."
                        )
                    else:
                        raise ValueError from e
            except (UnicodeDecodeError, ValueError):
                raise OSError(
                    "Unable to load weights from h5 file. "
                    "If you tried to load a TF 2.0 model from a PyTorch checkpoint, please set from_pt=True. "
                )

        if cls._keys_to_ignore_on_load_missing is not None:
            for pat in cls._keys_to_ignore_on_load_missing:
                missing_keys = [k for k in missing_keys if re.search(pat, k) is None]

        if cls._keys_to_ignore_on_load_unexpected is not None:
            for pat in cls._keys_to_ignore_on_load_unexpected:
                unexpected_keys = [k for k in unexpected_keys if re.search(pat, k) is None]

        if len(unexpected_keys) > 0:
            logger.warning(
                f"Some layers from the model checkpoint at {pretrained_model_name_or_path} were not used when"
                f" initializing {model.__class__.__name__}: {unexpected_keys}\n- This IS expected if you are"
                f" initializing {model.__class__.__name__} from the checkpoint of a model trained on another task or"
                " with another architecture (e.g. initializing a BertForSequenceClassification model from a"
                " BertForPreTraining model).\n- This IS NOT expected if you are initializing"
                f" {model.__class__.__name__} from the checkpoint of a model that you expect to be exactly identical"
                " (initializing a BertForSequenceClassification model from a BertForSequenceClassification model)."
            )
        else:
            logger.warning(f"All model checkpoint layers were used when initializing {model.__class__.__name__}.\n")

        if len(missing_keys) > 0:
            logger.warning(
                f"Some layers of {model.__class__.__name__} were not initialized from the model checkpoint at"
                f" {pretrained_model_name_or_path} and are newly initialized: {missing_keys}\nYou should probably"
                " TRAIN this model on a down-stream task to be able to use it for predictions and inference."
            )
        elif len(mismatched_keys) == 0:
            logger.warning(
                f"All the layers of {model.__class__.__name__} were initialized from the model checkpoint at"
                f" {pretrained_model_name_or_path}.\nIf your task is similar to the task the model of the checkpoint"
                f" was trained on, you can already use {model.__class__.__name__} for predictions without further"
                " training."
            )
        if len(mismatched_keys) > 0:
            mismatched_warning = "\n".join(
                [
                    f"- {key}: found shape {shape1} in the checkpoint and {shape2} in the model instantiated"
                    for key, shape1, shape2 in mismatched_keys
                ]
            )
            logger.warning(
                f"Some weights of {model.__class__.__name__} were not initialized from the model checkpoint at"
                f" {pretrained_model_name_or_path} and are newly initialized because the shapes did not"
                f" match:\n{mismatched_warning}\nYou should probably TRAIN this model on a down-stream task to be able"
                " to use it for predictions and inference."
            )

        # If it is a model with generation capabilities, attempt to load the generation config
        if model.can_generate():
            try:
                model.generation_config = GenerationConfig.from_pretrained(
                    pretrained_model_name_or_path,
                    cache_dir=cache_dir,
                    force_download=force_download,
                    resume_download=resume_download,
                    proxies=proxies,
                    local_files_only=local_files_only,
                    token=token,
                    revision=revision,
                    subfolder=subfolder,
                    _from_auto=from_auto_class,
                    _from_pipeline=from_pipeline,
                    **kwargs,
                )
            except OSError:
                logger.info(
                    "Generation config file not found, using a generation config created from the model config."
                )
                pass

        if output_loading_info:
            loading_info = {
                "missing_keys": missing_keys,
                "unexpected_keys": unexpected_keys,
                "mismatched_keys": mismatched_keys,
            }

            return model, loading_info

        return model

    def push_to_hub(
        self,
        repo_id: str,
        use_temp_dir: Optional[bool] = None,
        commit_message: Optional[str] = None,
        private: Optional[bool] = None,
        max_shard_size: Optional[Union[int, str]] = "10GB",
        token: Optional[Union[bool, str]] = None,
        # (`use_auth_token` is deprecated: we have to keep it here as we don't have **kwargs)
        use_auth_token: Optional[Union[bool, str]] = None,
        create_pr: bool = False,
        **base_model_card_args,
    ) -> str:
        """
        Upload the model files to the 🤗 Model Hub while synchronizing a local clone of the repo in `repo_path_or_name`.

        Parameters:
            repo_id (`str`):
                The name of the repository you want to push your model to. It should contain your organization name
                when pushing to a given organization.
            use_temp_dir (`bool`, *optional*):
                Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub.
                Will default to `True` if there is no directory named like `repo_id`, `False` otherwise.
            commit_message (`str`, *optional*):
                Message to commit while pushing. Will default to `"Upload model"`.
            private (`bool`, *optional*):
                Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists.
            token (`bool` or `str`, *optional*):
                The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
                when running `huggingface-cli login` (stored in `~/.huggingface`). Will default to `True` if `repo_url`
                is not specified.
            max_shard_size (`int` or `str`, *optional*, defaults to `"10GB"`):
                Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard
                will then be each of size lower than this size. If expressed as a string, needs to be digits followed
                by a unit (like `"5MB"`).
            create_pr (`bool`, *optional*, defaults to `False`):
                Whether or not to create a PR with the uploaded files or directly commit.

        Examples:

        ```python
        from transformers import TFAutoModel

        model = TFAutoModel.from_pretrained("google-bert/bert-base-cased")

        # Push the model to your namespace with the name "my-finetuned-bert".
        model.push_to_hub("my-finetuned-bert")

        # Push the model to an organization with the name "my-finetuned-bert".
        model.push_to_hub("huggingface/my-finetuned-bert")
        ```
        """
        if use_auth_token is not None:
            warnings.warn(
                "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
                FutureWarning,
            )
            if token is not None:
                raise ValueError(
                    "`token` and `use_auth_token` are both specified. Please set only the argument `token`."
                )
            token = use_auth_token

        if "repo_path_or_name" in base_model_card_args:
            warnings.warn(
                "The `repo_path_or_name` argument is deprecated and will be removed in v5 of Transformers. Use "
                "`repo_id` instead."
            )
            repo_id = base_model_card_args.pop("repo_path_or_name")
        # Deprecation warning will be sent after for repo_url and organization
        repo_url = base_model_card_args.pop("repo_url", None)
        organization = base_model_card_args.pop("organization", None)

        if os.path.isdir(repo_id):
            working_dir = repo_id
            repo_id = repo_id.split(os.path.sep)[-1]
        else:
            working_dir = repo_id.split("/")[-1]

        repo_id = self._create_repo(
            repo_id, private=private, token=token, repo_url=repo_url, organization=organization
        )

        if use_temp_dir is None:
            use_temp_dir = not os.path.isdir(working_dir)

        with working_or_temp_dir(working_dir=working_dir, use_temp_dir=use_temp_dir) as work_dir:
            files_timestamps = self._get_files_timestamps(work_dir)

            # Save all files.
            self.save_pretrained(work_dir, max_shard_size=max_shard_size)
            if hasattr(self, "history") and hasattr(self, "create_model_card"):
                # This is a Keras model and we might be able to fish out its History and make a model card out of it
                base_model_card_args = {
                    "output_dir": work_dir,
                    "model_name": Path(repo_id).name,
                }
                base_model_card_args.update(base_model_card_args)
                self.create_model_card(**base_model_card_args)

            self._upload_modified_files(
                work_dir,
                repo_id,
                files_timestamps,
                commit_message=commit_message,
                token=token,
                create_pr=create_pr,
            )

    @classmethod
    def register_for_auto_class(cls, auto_class="TFAutoModel"):
        """
        Register this class with a given auto class. This should only be used for custom models as the ones in the
        library are already mapped with an auto class.

        <Tip warning={true}>

        This API is experimental and may have some slight breaking changes in the next releases.

        </Tip>

        Args:
            auto_class (`str` or `type`, *optional*, defaults to `"TFAutoModel"`):
                The auto class to register this new model with.
        """
        if not isinstance(auto_class, str):
            auto_class = auto_class.__name__

        import transformers.models.auto as auto_module

        if not hasattr(auto_module, auto_class):
            raise ValueError(f"{auto_class} is not a valid auto class.")



SUMMARY:
Here is a concise summary of the provided code snippets:

This code snippet appears to be part of a library for handling Hugging Face models, particularly for TensorFlow (TF) and Keras. It provides methods for loading, saving, pushing, and registering TF/AutoModel instances with the model hub.

Specifically, it includes:

* `TFAutoModel.from_pretrained()`: Loads a pre-trained TF/TPU-AutoModel instance using the provided `model_name_or_path`.
* `push_to_hub()`: Pushes a model to the Hugging Face model hub.
* `register_for_auto_class()`: Registers this class as an auto-class for a given type.

Key methods and functions:

* `TFAutoModel.from_pretrained()`
* `push_to_hub()`
* `save_pretrained()`
* `_create_repo()`
* `_get_files_timestamps()`
* `_upload_modified_files()`

Notable features:

* Model hub integration
* TF/TPU-AutoModel support
* Automatic class registration

Overall, this code provides an interface for working with Hugging Face models and pushing them to the model hub.


