# embedding_metadata

key: chroma:document
string_value:
CODE:

def find_adapter_config_file(
    model_id: str,
    cache_dir: Optional[Union[str, os.PathLike]] = None,
    force_download: bool = False,
    resume_download: Optional[bool] = None,
    proxies: Optional[Dict[str, str]] = None,
    token: Optional[Union[bool, str]] = None,
    revision: Optional[str] = None,
    local_files_only: bool = False,
    subfolder: str = "",
    _commit_hash: Optional[str] = None,
) -> Optional[str]:
    r"""
    Simply checks if the model stored on the Hub or locally is an adapter model or not, return the path of the adapter
    config file if it is, None otherwise.

    Args:
        model_id (`str`):
            The identifier of the model to look for, can be either a local path or an id to the repository on the Hub.
        cache_dir (`str` or `os.PathLike`, *optional*):
            Path to a directory in which a downloaded pretrained model configuration should be cached if the standard
            cache should not be used.
        force_download (`bool`, *optional*, defaults to `False`):
            Whether or not to force to (re-)download the configuration files and override the cached versions if they
            exist.
        resume_download:
            Deprecated and ignored. All downloads are now resumed by default when possible.
            Will be removed in v5 of Transformers.
        proxies (`Dict[str, str]`, *optional*):
            A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
            'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
        token (`str` or *bool*, *optional*):
            The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
            when running `huggingface-cli login` (stored in `~/.huggingface`).
        revision (`str`, *optional*, defaults to `"main"`):
            The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
            git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
            identifier allowed by git.

            <Tip>

            To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>".

            </Tip>

        local_files_only (`bool`, *optional*, defaults to `False`):
            If `True`, will only try to load the tokenizer configuration from local files.
        subfolder (`str`, *optional*, defaults to `""`):
            In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
            specify the folder name here.
    """
    adapter_cached_filename = None
    if model_id is None:
        return None
    elif os.path.isdir(model_id):
        list_remote_files = os.listdir(model_id)
        if ADAPTER_CONFIG_NAME in list_remote_files:
            adapter_cached_filename = os.path.join(model_id, ADAPTER_CONFIG_NAME)
    else:
        adapter_cached_filename = cached_file(
            model_id,
            ADAPTER_CONFIG_NAME,
            cache_dir=cache_dir,
            force_download=force_download,
            resume_download=resume_download,
            proxies=proxies,
            token=token,
            revision=revision,
            local_files_only=local_files_only,
            subfolder=subfolder,
            _commit_hash=_commit_hash,
            _raise_exceptions_for_gated_repo=False,
            _raise_exceptions_for_missing_entries=False,
            _raise_exceptions_for_connection_errors=False,
        )



SUMMARY:
**Summary:**

The `find_adapter_config_file` function retrieves the path to an adapter configuration file based on a given model ID. It checks if the model is stored locally or on the Hub, and if it's an adapter model, returns the cached filename.

**Key Functionality:**

* Checks for local/Hub model with ID
* Returns cached adapter config file name (if present)

**Implementation Details:**

* `os.path.isdir(model_id)` checks for local directory
* `ADAPTER_CONFIG_NAME` check in local files for matches
* `cached_file()` function downloads/retrieves configuration file using cache and force options

**Notable Dependencies/Requirements:**

* `ADAPTER_CONFIG_NAME`: defined elsewhere, likely a constant or variable within the module
* `os.PathLike`, `Optional[Union[str, ...]]` types used for parameters and return value


key: chroma:document
string_value:
CODE:

def check_peft_version(min_version: str) -> None:
    r"""
    Checks if the version of PEFT is compatible.

    Args:
        version (`str`):
            The version of PEFT to check against.
    """
    if not is_peft_available():
        raise ValueError("PEFT is not installed. Please install it with `pip install peft`")

    is_peft_version_compatible = version.parse(importlib.metadata.version("peft")) >= version.parse(min_version)

    if not is_peft_version_compatible:
        raise ValueError(
            f"The version of PEFT you are using is not compatible, please use a version that is greater"
            f" than {min_version}"


SUMMARY:
**Summary: PEFT Version Compatibility Checker**

### Purpose and Functionality

Checks if the installed version of PEFT meets or exceeds the specified minimum compatibility version.

### Implementation Details

*   Compares the installed PEFT version with the minimum required version using `importlib.metadata.version("peft")`.
*   Raises a ValueError if the installed version is incompatible.
*   Verifies that PEFT is available and installed before comparisons.

### Key Method Signature:

`is_peft_available() -> bool`

### Notable Dependency:

*   `importlib.metadata.version("peft")`: Retrieves the installed version of PEFT.


key: chroma:document
string_value:
CODE:

def _generate_supported_model_class_names(
    model_name: Type[PretrainedConfig],
    supported_tasks: Optional[Union[str, List[str]]] = None,
) -> List[str]:
    task_mapping = {
        "default": MODEL_MAPPING_NAMES,
        "pretraining": MODEL_FOR_PRETRAINING_MAPPING_NAMES,
        "next-sentence-prediction": MODEL_FOR_NEXT_SENTENCE_PREDICTION_MAPPING_NAMES,
        "masked-lm": MODEL_FOR_MASKED_LM_MAPPING_NAMES,
        "causal-lm": MODEL_FOR_CAUSAL_LM_MAPPING_NAMES,
        "seq2seq-lm": MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES,
        "speech-seq2seq": MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES,
        "multiple-choice": MODEL_FOR_MULTIPLE_CHOICE_MAPPING_NAMES,
        "document-question-answering": MODEL_FOR_DOCUMENT_QUESTION_ANSWERING_MAPPING_NAMES,
        "question-answering": MODEL_FOR_QUESTION_ANSWERING_MAPPING_NAMES,
        "sequence-classification": MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING_NAMES,
        "token-classification": MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMES,
        "masked-image-modeling": MODEL_FOR_MASKED_IMAGE_MODELING_MAPPING_NAMES,
        "image-classification": MODEL_FOR_IMAGE_CLASSIFICATION_MAPPING_NAMES,
        "zero-shot-image-classification": MODEL_FOR_ZERO_SHOT_IMAGE_CLASSIFICATION_MAPPING_NAMES,
        "ctc": MODEL_FOR_CTC_MAPPING_NAMES,
        "audio-classification": MODEL_FOR_AUDIO_CLASSIFICATION_MAPPING_NAMES,
        "semantic-segmentation": MODEL_FOR_SEMANTIC_SEGMENTATION_MAPPING_NAMES,
        "backbone": MODEL_FOR_BACKBONE_MAPPING_NAMES,
        "image-feature-extraction": MODEL_FOR_IMAGE_MAPPING_NAMES,
    }

    if supported_tasks is None:
        supported_tasks = task_mapping.keys()
    if isinstance(supported_tasks, str):
        supported_tasks = [supported_tasks]

    model_class_names = []
    for task in supported_tasks:
        class_name = task_mapping[task].get(model_name, None)
        if class_name:
            model_class_names.append(class_name)



SUMMARY:
**Function Summary**

`_generate_supported_model_class_names`

Generates a list of supported model class names for a given pre-trained model and specified tasks.

**Purpose**
Generates class names for supported models based on provided `model_name` and `supported_tasks`.

**Key Implementation Details**

* Mapping between model types (`task_mapping`) and class names
* Checks if `supported_tasks` is None, defaulting to all task mapping keys
* Iterates over each task in `supported_tasks`, looking up matching class name in `task_mapping`
* Returns a list of supported model class names

**Notable Dependencies/Requirements**

* `Type[PretrainedConfig]`: Type hint for `model_name` parameter
* `Optional[Union[str, List[str]]]`: Type hint for `supported_tasks` parameter
* `List[str]`: Return type hint for function result


key: chroma:document
string_value:
CODE:

def torch_nn_embedding(self, input):


SUMMARY:
**Summary**

```python
torch_nn_embedding(self, input)
-----------------

Creates a new tensor with shape {self.weight.shape[:-1]} x input.shape[1] x {self.weight.shape[-1]}, initialized with meta device and dtype matching self.weight.

Key implementation details:

- Uses `torch.empty` to create an empty tensor.
- Sets device to "meta" and dtype to match weight's dtype.
- Assumes self.weight is a constant used in the embedding layer.
```

**Notable dependencies and requirements**

- **torch.nn.Embedding**: Serves as a conceptual basis for this function, utilizing its internal implementation details.
- `self.weight`: A constant tensor with known shape and data type.


key: chroma:document
string_value:
CODE:

def torch_nn_functional_embedding(
    input, weight, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False
):


SUMMARY:
**torch_nn_functional_embedding Function Summary**

* **Purpose**: Creates an embedding tensor with the same shape as the input tensor and the last dimension from the `weight` tensor.

**Key Details**:

* Takes input tensor, weight matrix, and optional parameters (`padding_idx`, `max_norm`, `norm_type`, `scale_grad_by_freq`, `sparse`)
* Returns an empty tensor with dimensions `input.shape + (weight.shape[-1],)` on the "meta" device
* Uses dtype of `weight` tensor

**Implementation Note**: The function uses torch's empty tensor constructor to create a new tensor without initializing its values.


key: chroma:document
string_value:
CODE:

def torch_nn_layernorm(self, input):


SUMMARY:
**Summary:**

The `torch_nn_layernorm` method is a placeholder that simply returns the original input without applying any normalization. It appears to be a stub function for TorchNeuralNetwork implementation.

**Key Implementation Details:**

- No layer normalization algorithm is applied (returns input unchanged)
- Does not implement TorchNeuralNetwork layer normalization logic

**Notable Dependencies/Requirements:**

- None explicitly mentioned, but it's likely that this function relies on the `torch.nn` module, specifically `LayerNorm`.

This summary aims to capture the essence of the code chunk in a concise manner while highlighting the missing implementation details.


key: chroma:document
string_value:
CODE:

def torch_nn_groupnorm(self, input):


SUMMARY:
**Summary:**

**Function Name:** `torch_nn_groupnorm`

**Purpose:** Applies Group Normalization to the input tensor in PyTorch.

**Primary Purpose:** Normalization layer using group-wise mean and variance calibration.

**Implementation Details:**

* Uses the PyTorch neural network module.
* No explicit algorithm is implemented; it simply returns the original input, implying that the actual execution of Group Normalization occurs outside this function (e.g., in a parent class or elsewhere).

**Notable Dependencies/Requirements:**

* `torch`: PyTorch library (version not specified).
* Assuming inherited interfaces from PyTorch's neural network modules.


key: chroma:document
string_value:
CODE:

def torch_nn_linear(self, input):


SUMMARY:
**Summary:**

* `torch.nn.Linear` subclass with custom initialization.
* Creates a new PyTorch tensor initialized with zeros, mirroring the structure of a standard neural network linear layer.
* Returns a tensor of shape `(batch_size, input_dim, out_features)` on the device "meta".

Key details:

* Input dimension: `input.shape[:-1]`
* Output dimension: `self.out_features`
* Device: `"meta"`

Note: This function appears to be part of a neural network class, possibly for meta-learning or other specific applications.


key: chroma:document
string_value:
CODE:

def torch_relu(x):


SUMMARY:
**Summary:**

The `torch_relu` function is a simple wrapper around the tensor value `x`, performing a rectified linear unit (ReLU) activation operation.

* Primary purpose: Applies ReLU to input tensor `x`.
* Key implementation detail: Wraps the input `x` without any additional computation.
* Notable dependencies:
	+ None explicit, as it relies on PyTorch's tensor operations.


key: chroma:document
string_value:
CODE:

def torch_nn_relu(self, x):


SUMMARY:
**Summary:**

The `torch_nn_relu` function takes an input `x`. It returns the ReLU activation function applied to `x`. Notably, this implementation seems incomplete as it simply returns `x` without any actual ReLU activation. In typical implementations, one would expect to see a return value of `'max(0, x)'`.

If we assume this is intended to be a placeholder or an error state for the full implementation:

**Key points:**

* Implementation purpose and functionality ( incomplete ): Applies ReLU activation function.
* Key algorithm or design choice: None (or expected but missing).
* Notable dependencies or requirements:
	+ External dependency: `torch` library, specifically its neural network module.
	+ Inherited or implemented interfaces: `'max(0, x)'` is a placeholder for the complete activation function.


key: chroma:document
string_value:
CODE:

def torch_nn_functional_relu(x, inplace=False):
    if not inplace:
        raise ValueError("Don't support in-place functional.relu for MetaTensor analysis")


SUMMARY:
**Code Summary:**

### `torch_nn_functional_relu` Function

**Purpose:** Applies the ReLU activation function to input tensor `x`.

**Key Implementation Details:**

- **Activation Function:** Computes the ReLU (Rectified Linear Unit) of each element in the input tensor.
- **In-place Mode:** Activates input tensor in-place if `inplace=True`, otherwise returns a new output tensor.

**Dependencies and Requirements:**

- `torch.nn.functional.relu` module is imported, but not explicitly used due to function re-definition.


key: chroma:document
string_value:
CODE:

def torch_where(condition, x, y):
    # torch.where returns the broadcasted tensor of condition, x, and y,
    # so hack it by using addition


SUMMARY:
**Summary:**

This function, `torch_where`, overrides PyTorch's built-in `torch.where` with a custom implementation. It replaces the original tensor values with broadcasted versions of `x`.

*   **Primary Purpose:** Replaces tensor values based on condition `condition` and returns a new tensor.
*   **Key Implementation Details:**
    *   Uses addition to mimic broadcasting behavior.
    *   Returns the result for the `meta` device.
*   **Notable Dependencies:** None specified, relying on PyTorch's built-in data structures.

This custom implementation suggests that the author wants more control over how tensor values are updated based on conditions. If further analysis is needed, consider examining any errors or edge cases in this function.


key: chroma:document
string_value:
CODE:

def torch_abs(input, *, out=None):
    if out is not None:
        raise ValueError("Don't support in-place abs for MetaTensor analysis")


SUMMARY:
**torch_abs Function Summary**

This function calculates the absolute value of its input `input` and returns it. It serves as a wrapper for existing tensor operations, ensuring consistency across various libraries.

* **Key Implementation Details:**
	+ The `out=None` parameter is ignored, indicating that this function does not perform in-place modifications.
	+ The function relies on the underlying tensor library to compute the absolute value.
* **Main Purpose and Functionality:** Returns the absolute value of the input tensor.
* **Notable Dependencies or Requirements:**
	- torch (Tensor Library)
 
   This code chunk can be summarized as:
```
torch_abs(input, out=None) -> Returns |input|
    Raises ValueError on in-place abs for MetaTensor analysis
```


key: chroma:document
string_value:
CODE:

def torch_arange(*args, **kwargs):
    n = len(args)
    step = 1
    if n == 1:
        start = 0
        end = args[0]
    elif n == 2:
        start, end = args
    else:
        start, end, step = args
    if isinstance(start, float):
        start = int(start)
    if isinstance(end, float):
        start = int(end)
    if isinstance(step, float):
        step = int(step)
    step = kwargs.get("step", step)
    dtype = kwargs.get("dtype")


SUMMARY:
**torch_arange function:**

*   Purpose:
    *   Creates an array of `start` stepped increments up to and including `end`.
    *   Utilizes the PyTorch library for efficient tensor creation.
*   Parameters:

    *   `n` arguments (variable length): specifies the number of arguments passed, handling single, two-element, or three-element sequences.
        *   In a three-element sequence: start, end, step
    *   Arbitrary keyword arguments (`kwargs`):
        *   `step`: sets the step size (default value taken from the first command line argument).
        *   `dtype`: specified type of elements in the tensor (default=auto-determined).
*   Implementation:
    *   Converts input types to integers where applicable.
    *   If no step value is provided via keyword arguments, uses the default values stored within the function's defined sequence parameters (`args` or kwargs with "step").
*:   Returns an empty PyTorch tensor of a size specified by the difference between start and end divided by the step.


key: chroma:document
string_value:
CODE:

def torch_full(*args, **kwargs):
    args = list(args)
    # We set the fill value to 1 as its value is not important as long as it's not a tensor on the `meta` device.
    if len(args) > 1:
        args[1] = 1
    else:
        kwargs["fill_value"] = 1
    kwargs_without_device = dict(kwargs)
    kwargs_without_device.pop("device", None)


SUMMARY:
**torch_full Function Summary**

**Purpose**: Fills a tensor with a specified fill value.

**Parameters:**

* `fill_value`: The fill value for the tensor.
* `device`: (Optional) The device to allocate the tensor on. Defaults to `"meta"` if not provided or raises an exception when given a valid device other than `"meta"` and `"cuda"`.


**Implementation Details:**

* If multiple arguments are provided, the second argument is set to `1` as its value.
* If only a single keyword argument is provided, it's used for filling. However, if no fill value is specified or provided (except the `device` or the one of the positional args), raises an exception when given `"meta"`.

**Note:** This function utilizes `torch(full)` which fills tensor with value from kwarg that has 'fill_value'.


key: chroma:document
string_value:
CODE:

def torch_cat(tensors, dim=None, axis=None, *, out=None):
    if dim is None and axis is None:
        dim = 0
    if dim is None and axis is not None:
        dim = axis
    if dim < 0:
        dim = tensors[0].dim() + dim
    shapes = [t.shape for t in tensors]
    shape = list(shapes[0])
    concatenated_dim = sum(shape[dim] for shape in shapes)
    final_shape = shape[:dim] + [concatenated_dim] + shape[dim + 1 :]


SUMMARY:
**torch_cat Function Summary**

The `torch_cat` function concatenates multiple tensors along a specified dimension. It takes in:

- `tensors`: the list of tensors to concatenate
- `dim`: the dimension to concatenate (optional, defaults to 0)
- `axis`: the axis to concatenate (optional, only valid if no `dim` is specified)

It returns a new tensor with a concatenated shape defined by the specified dimension.

**Notable Implementation Details:**

- Uses PyTorch's default behavior when `dim` and `axis` are both None, setting `dim` to 0.
- Ensures that `dim` is non-negative by adding it to the first tensor's dimension if necessary.

**Dependent Libraries:**

- PyTorch (imported implicitly)
- torch.empty for creating a new tensor of the specified shape

This function leverages PyTorch's tensor creation and manipulation capabilities to efficiently concatenate tensors along a specified dimension.


key: chroma:document
string_value:
CODE:

def torch_stack(tensors, dim=None, axis=None, *, out=None):
    if dim is None and axis is None:
        dim = 0
    if dim is None and axis is not None:
        dim = axis
    if dim < 0:
        dim = tensors[0].dim() + 1 + dim
    shape = list(tensors[0].shape)
    shape.insert(dim, len(tensors))


SUMMARY:
**Function Summary: `torch_stack`**

The `torch_stack` function stacks multiple tensors along a specified dimension using PyTorch. It takes in:

* `tensors`: the input tensor list
* `dim`: the dimension to stack (optional)
* `axis`: an alternative axis specification (if `dim` is not provided)
* `out`: output tensor (default: None)

It creates a new tensor with the specified shape, where the specified dimension is extended with the number of tensors. The function returns a PyTorch tensor on the "meta" device.

**Implementation Details:**

* Uses `torch.empty` to create a new tensor on the "meta" device
* Calculates the new shape using list indexing and insertion

**Dependencies and Requirements:**

* PyTorch library
* Requires access to a PyTorch tensor device (in this case, "meta")


key: chroma:document
string_value:
CODE:

def torch_add(input, other, *, alpha=1, out=None):
    if not isinstance(input, torch.Tensor):
        return torch.empty_like(other, device="meta")
    if not isinstance(other, torch.Tensor):
        return torch.empty_like(input, device="meta")
    max_length = max(input.dim(), other.dim())
    input_shape = list(input.shape) + [1] * (max_length - input.dim())
    other_shape = list(other.shape) + [1] * (max_length - other.dim())
    shape = []
    for i in range(max_length):
        shape.append(max(input_shape[i], other_shape[i]))


SUMMARY:
**Summary**

This is an implementation of a custom `torch_add` function that concatenates two input tensors along a dynamic axis. The main purpose is to create a new tensor with the same shape as the longest input tensor, filled with zeros.

**Key Details**

* Function signature: `torch_add(input, other, *, alpha=1, out=None)`
* Purpose: Concatenates `input` and `other` tensors along a dynamic axis.
* Key algorithm:
	+ Fills output tensor with ones using the formula `(input + other) / alpha`.
* Notable dependencies:
	+ `torch.Tensor`
	+ `device="meta"`

**Implementation Choices**

* Uses `torch.empty` to create an empty output tensor with the maximum shape.
* Fills output tensor using a formula that combines `alpha`, `input`, and `other`.
* Does not accept an existing output tensor (`out=None`).


key: chroma:document
string_value:
CODE:

def torch_mul(input, other, *, out=None):


SUMMARY:
**Summary:**

This is a Python function `torch_mul` that multiplies two inputs component-wise using the PyTorch library.

*   **Primary purpose:** Computes element-wise multiplication of `input` and `other`.
*   **Key features:**
    *   Uses PyTorch's `torch_add` function to perform addition.
    *   Applies the `out=None` positional argument to delegate output handling.
*   **Dependencies:** None
*   **Relationship:** This function relies on existing functionality in the PyTorch library.


key: chroma:document
string_value:
CODE:

def torch_tensor_mul(self, other):


SUMMARY:
**Summary:**

This function, `torch_tensor_mul`, is a simplified wrapper for performing tensor multiplication between two input tensors. It calls the `torch_mul` function, but does not provide any additional functionality or caching for repeated calculations.

**Key Points:**

* Simplified wrapper around `torch_mul`
* No caching or optimization implemented

(Note: The actual implementation of `torch_mul` is not shown here as it's likely included in a larger library or framework)


key: chroma:document
string_value:
CODE:

def torch_matmul(input, other, *, out=None):
    d1 = input.dim()
    d2 = other.dim()
    shape = None
    if d1 == 1 and d2 == 1:
        shape = None
    elif d1 == 2 and d2 == 2:
        shape = (input.size(0), other.size(1))
    elif d1 == 1 and d2 == 2:
        shape = (other.size(1),)
    elif d1 == 2 and d1 == 1:
        shape = (input.size(0),)
    else:
        max_length = max(input.dim(), other.dim())
        shape1 = list(input.shape)
        shape2 = list(other.shape)
        if d1 == 1:
            shape1 = [1] + shape1
        if d2 == 1:
            shape2.append(1)
        shape1 = [-1] * (max_length - d1) + list(input.shape)
        shape2 = [-1] * (max_length - d2) + list(other.shape)
        shape = []
        for i in range(max_length):
            shape.append(max(shape1[i], shape2[i]))
        shape[-2] = shape1[-2]
        shape[-1] = shape2[-1]
        if d1 == 1:
            shape.pop(-2)
        if d2 == 1:
            shape.pop(-1)
    if shape is None:
        return torch.tensor(0.0, device="meta")


SUMMARY:
**Function Summary: `torch_matmul`**

**Purpose and Functionality:**
The `torch_matmul` function performs matrix multiplication between two input tensors (`input` and `other`) with optional output tensor specification. It checks dimensionality to determine the appropriate output shape.

**Key Implementation Details:**

* Supports 1x1, 2x2, and higher-dimensional matrix multiplications.
* Handles batch dimensions using PyTorch's indexing syntax (e.g., `input.size(0)`).
* For non-square matrices, adds missing dimensions to match the larger matrix.

**Notable Dependencies/Requirements:**

* Input tensors (`input` and `other`) are expected to be PyTorch tensors.
* External dependencies:
	+ PyTorch tensor operations (e.g., `torch.empty`, `torch.tensor`)
	+ PyTorch dimension indexing syntax (e.g., `input.size(0)`)

**Method Signatures and Interfaces:**

* Input parameters:
	+ `input`: input tensor
	+ `other`: input tensor
	+ `out`: optional output tensor specification (default: None)
* Output returns:
	+ A newly allocated PyTorch tensor with the specified shape, or a default value if shape is Not provided.


key: chroma:document
string_value:
CODE:

def torch_bmm(input, mat2, *, out=None):
    if out is not None:
        raise ValueError("Don't support in-place bmm for MetaTensor analysis")
    batch_size, n, m = input.shape
    _, _, p = mat2.shape


SUMMARY:
**Summary:**

The `torch_bmm` function performs broadcasted matrix multiply (bMM) operation between two tensors (`input` and `mat2`) with shape `(batch_size, n, m)` and `(n, p)`, respectively. It creates an output tensor of shape `(batch_size, n, p)` on the "meta" device. Raises an error when in-place mode is required.

**Key implementation details:**

* Algo: Broadcasted matrix multiplication
* Design choice: No support for in-place execution (error raised)
* Method signature: `torch_bmm(input, mat2, *, out=None)`

**Notable dependencies and requirements:**

* `torch`: PyTorch library
* `MetaTensor analysis` context (~assuming a specific scientific computing or research context)
* `pytorch.empty`: PyTorch function to create a new tensor


key: chroma:document
string_value:
CODE:

def torch_baddbmm(input, batch1, batch2, *, beta=1, alpha=1, out=None):
    if out is not None:
        raise ValueError("Don't support in-place baddbmm for MetaTensor analysis")


SUMMARY:
**torch_baddbmm Function Summary**

### Purpose and Functionality
The `torch_baddbmm` function performs a batched cross-batch matrix product using the batch-batch (baddb) algorithm from PyTorch. It takes in two inputs (`batch1` and `batch2`) and computes their product, with optional scaling factors (`beta` and `alpha`). The result is returned directly.

### Key Implementation Details

* Uses the `torch_bmm` function internally
* Sets `out=None` to disallow in-place operations for MetaTensor analysis

### Dependencies/Requirements

* torch ( PyTorch library)

### Interface/Distinguished Parameters

* batch1 and batch2: input tensors for matrix product
* beta and alpha: optional scaling factors (default=1)
* out: output tensor (not supported for in-place operations)


key: chroma:document
string_value:
CODE:

def torch_tensor_baddbmm(self, batch1, batch2, *, beta=1, alpha=1, out=None):


SUMMARY:
**Summary:**

This function, `torch_tensor_baddbmm`, appears to be a wrapper or alias for the `torch_baddbmm` functionality. It calls the external `torch_baddbmm` function with identical arguments (`batch1`, `batch2`, `beta=beta`, `alpha=alpha`, and `out=out`) prefixed with `self`. The primary purpose is to maintain consistency with a decorator or mixin pattern, but it doesn't add any new functionality.

**Summary of key implementation details:**

- Wraps the external `torch_baddbmm` function
- Passes arguments by prefixing 'self' (indicating a class context)
- Does not alter any algorithmic behavior

**Notable dependencies and requirements:**

- External dependency on `torch_baddbmm`
- Uses `beta`, `alpha`, and `out` parameters
- Imposed interface for calling the external function (`torch_baddbmm`)


key: chroma:document
string_value:
CODE:

def torch_einsum(equation, *operands):
    # TODO: infer shape without performing the computation, this might be quite hard.
    concrete_operands = (torch.empty_like(operand, device="cpu") for operand in operands)


SUMMARY:
**Summary**

`torch_einsum`: A function implementing the Torch tensor contraction, specified by `equation` with optional input `operands`. 

**Key Features**

- Applies Einsum contraction with specified equation and operands.
- Returns the resulting tensor on device "meta".
 
**Important Details**

- Uses `torch.empty_like` for operand creation to optimize computations.
- Utilizes `torch.einsum` core library function.

**Interfaces/Inheritance**

- Extends Torch functionality, utilizing `torch.einsum`.
- None.


key: chroma:document
string_value:
CODE:

def torch_tensor_repeat(self, *sizes):
    shape = list(self.shape)
    for i, x in enumerate(sizes):
        shape[i] *= x


SUMMARY:
**Code Summary**

`torch_tensor_repeat`: Repeats the tensor along specified dimensions and returns a new tensor with the updated shape.

**Key Details**

- Multiplies each dimension size by its corresponding input size from `sizes`.
- Returns an empty tensor of the new calculated shape on the "meta" device.
- Utilizes `len(self.shape) + 1` for indexing due to variable length argument.

**Dependencies**

*torch*: The PyTorch library for deep learning.
* meta: The target device for the new tensor (specific device not explicitly defined).


key: chroma:document
string_value:
CODE:

def torch_repeat_interleave(*args, dim=None, output_size=None):
    num_args = len(args)
    if num_args == 1:
        shape = [output_size if output_size is not None else args[0].sum()]
    else:
        shape = list(args[0].shape)
        if dim is None:
            if num_args > 2:
                dim = args[2]
            else:
                shape = [sum(shape)]
                dim = 0
        repeats = args[1]
        if isinstance(repeats, int) or torch.numel(repeats) == 1:
            shape[dim] *= int(repeats)
        else:
            shape[dim] = output_size if output_size is not None else repeats.sum()


SUMMARY:
**Summary:**

Returns a tensor with the specified dimensions and repeated along the specified dimension(s).

* **Purpose:** Performs tuple repetition ( torch.repeat_interleave equivalent) in PyTorch.
* **Primary purpose:** Repeat elements of one tensor along multiple tensors using one-dimension indexing.
* **Key implementation details:**
	+ Args: `repeats` and `dim` control the repeat factor and dimension respectively.
	+ Algorithm: Repeats along a specific dimension if specified, otherwise uses default behavior.

**Notable Dependencies and Requirements:**

*   PyTorch library (`torch`)
*   Built-in Python types and functions (e.g., `int`, `list`, etc.)


key: chroma:document
string_value:
CODE:

def torch_index_select(input, dim, index, *, out=None):
    shape = list(input.shape)
    shape[dim] = len(index)


SUMMARY:
**Tensor Index Select Function Summary**

* **Main Purpose:** Creates a tensor with dimensions increased along the specified dimension (`dim`) to match the length of input indices.

**Key Implementation Details:**
 
- Increases shape in `dim` dimension using `shape = list(input.shape)` and `shape[dim] = len(index)`
 

**Notable Dependencies or Requirements:**

* Uses PyTorch tensor (`torch.empty`)
* Requires explicit `device` specification (`"meta"`)


key: chroma:document
string_value:
CODE:

def torch_tensor_index_select(self, dim, index):


SUMMARY:
**Summary:**

### Function Purpose and Functional Details

The `torch_tensor_index_select` method is an overridden implementation of the parent class's method. It takes in two parameters, `dim` and `index`, and returns the result of calling the base class method with the same parameter values.

In essence, this method does not alter the original functionality of its parent class but instead acts as a passthrough, delegating all logic to the base method.

### Notable Implementation Details

* The method uses a shortcut by directly forwarding calls to the parent class's implementation (`torch_index_select(self, dim, index)`).

### Dependencies and Requirements

* `torch` is referenced, indicating this function relies on PyTorch for tensor operations.
* A parent class, likely `Model`, has an overridden version of `torch_index_select`.


key: chroma:document
string_value:
CODE:

def torch_gather(input, dim, index, *, sparse_grad=False, out=None):
    shape = list(input.shape)
    shape[dim] = index.shape[dim]


SUMMARY:
**torch_gather Function Summary**

Creates a tensor by gathering scalar values from input along specified dimension using provided indices.

### Key Details:

* Primary purpose: Gather elements from `input` based on `index` in the specified dimension (`dim`) using PyTorch tensor operations.
* Implemented PyTorch operation: `torch.gather`
* Return type: A new, temporarily allocated PyTorch tensor with shape adapted to accommodate gathered values.

### Implementation Insights:

* The function modifies input shapes for dim=0, replacing it with index dimensions, indicating a sparse gathering setup.
* Output is returned as an empty, device-specific tensor "meta" using `torch.empty`.


key: chroma:document
string_value:
CODE:

def torch_tensor_gather(self, dim, index):


SUMMARY:
**Summary**

`torch_tensor_gather` method substitutes `torch.gather(self, dim, index)`.

* **Implementation**: Directly calls the `torch.gather` function with custom object (`self`) as target tensor.
* **Key insight**: Does not implement algorithm specific to the class, but delegates to built-in functionality of PyTorch's `torch.gather`. 
* **Depends on**:
	+ None
*


key: chroma:document
string_value:
CODE:

def torch_roll(input, shifts, dims=None):


SUMMARY:
**torch_roll Function Summary**

*   **Purpose**: This function applies circular tensor roll to the input tensor along the specified dimensions. It returns the original input without modification if no dimensions are provided.

*   **Key Implementation Details**:
    *   The `torch_roll` function does not perform any actual rolling operations on the input tensor; instead, it simply returns the original input.
    *   If `dims` are not specified or empty, the operation has no effect.

*   **Notable Dependencies/Requirements**: None

*   **Method Signature and Interfaces**: The function consists of a single method:

    ```python
def torch_roll(input, shifts, dims=None):
```

    No specific interfaces are implemented.


key: chroma:document
string_value:
CODE:

def torch_flip(input, dims):


SUMMARY:
**Summary**

### `torch_flip`

**Purpose**: Reverses the order of dimensions in a PyTorch tensor.

**Implementation Details**:

* Repeats each dimension twice without modifying the data itself.

### Notable Dependencies/Requirements

* PyTorch library (not explicitly referenced, but implied as the context)

### Method Signature
```
torch_flip(input: torch.Tensor, dims: List[int])
```


key: chroma:document
string_value:
CODE:

def torch_tensor_flip(self, dims):


SUMMARY:
**Summary:**

The `torch_tensor_flip` function is not implemented and simply returns the current instance of a class derived from it (`self`). 

**Underscored Analysis Points:

* **Primary Purpose:** The function appears to be intended for implementing tensor flipping functionality using PyTorch.
* **Key Design Choice:** The current implementation does not actually flip tensors, suggesting a placeholder or incomplete function. 
* **Dependent Class/Interface:** The function relies on the instance of `self` being an object that possesses additional methods or attributes for tensor manipulation.

**Related Details:**

- No implementation is provided for flipping PyTorch tensors.
- Returns `self`, implying potential method chaining or customization in a class context.


key: chroma:document
string_value:
CODE:

def torch_nn_conv1d(self, input):
    l_in = input.shape[-1]
    shape = None
    padding = self.padding
    if padding == "valid":
        padding = (0, 0)
    if padding == "same":
        shape = list(input.shape)
    if shape is None:
        shape = list(input.shape)
        l_out = math.floor(
            (l_in + 2 * padding[0] - self.dilation[0] * (self.kernel_size[0] - 1) - 1) / self.stride[0] + 1
        )
        shape[-1] = l_out
    shape[-2] = self.out_channels


SUMMARY:
**Summary**

This function implements a `torch_nn_conv1d` layer. It computes the output shape of a 1D convolutional layer given the input shape and configuration.

**Main Purpose and Functionality**
 
- Calculates the output shape of a 1D convolutional layer (`torch.nn.Conv1d`) based on the provided input shape, padding type, dilation rate, kernel size, and stride.
- Returns an empty tensor with the calculated shape, optimized for meta training by setting device to "meta".

**Key Implementation Details**
 
- Uses mathematical formulas to calculate the output shape:
  - `l_out = math.floor((l_in + 2 * padding[0] - self.dilation[0] * (self.kernel_size[0] - 1) - 1) / self.stride[0] + 1)`
  - Sets the last dimension of the input shape to the calculated output length (`shape[-1] = l_out`).
- Handles three types of padding: "valid", "same", and does not require any padding when both are set (as in this case where it's explicitly assigned a value).

**Notable Dependencies or Requirements**
 
- Utilizes `torch.nn.Conv1d`
- Requires `math.floor`
- Uses `torch.empty` to create an empty tensor with the calculated shape.
- Relies on device configuration for "meta" training.


key: chroma:document
string_value:
CODE:

def torch_nn_conv2d(self, input):
    h_in, w_in = input.shape[-2:]
    shape = None
    padding = self.padding
    if padding == "valid":
        padding = (0, 0)
    if padding == "same":
        shape = list(input.shape)
    if shape is None:
        shape = list(input.shape)
        h_out = math.floor(
            (h_in + 2 * padding[0] - self.dilation[0] * (self.kernel_size[0] - 1) - 1) / self.stride[0] + 1
        )
        w_out = math.floor(
            (w_in + 2 * padding[1] - self.dilation[1] * (self.kernel_size[1] - 1) - 1) / self.stride[1] + 1
        )
        shape[-2:] = [h_out, w_out]
    shape[-3] = self.out_channels


SUMMARY:
**Summary: `torch_nn_conv2d` Function**

Constructs and returns a Torch tensor representing the output of a 2D convolutional layer.

**Primary Purpose:**
This function generates an output tensor of shape `(height_out, width_out, channels)`, where:

* `height_out` and `width_out` are calculated based on input dimensions and kernel parameters.
* `channels` is set to `self.out_channels`.

**Key Implementation Details:**

* Calculates `height_out` and `width_out` using formulas derived from convolutional layer's spatial padding and dilation.
* Employs Torch's `empty()` function to create an empty tensor, specifying the desired device ("meta").

**Notable Dependencies/Requirements:**

* `self.padding`: Kernel padding type (e.g., "valid" or "same").
* `self.dilation`: Spatial dilation factor for convolutional layer.
* `self.kernel_size`: Kernel size tuple `(height, width)`.
* `self.stride`: Convolution stride tuple `(height, width)`.
* `torch.empty()`, `math.floor()`.


key: chroma:document
string_value:
CODE:

def torch_squeeze(input, dim=None):
    shape = list(input.shape)
    if dim is not None:
        if dim < 0:
            dim = input.dim() + dim
        if shape[dim] == 1:
            shape.pop(dim)
    else:
        new_shape = []
        for dim_value in shape:
            if dim_value == 1:
                continue
            new_shape.append(dim_value)
        shape = new_shape


SUMMARY:
**Summary:**

The `torch_squeeze` function reduces the dimension size of a PyTorch tensor along specified axis by replacing consecutive zeros with zero padding. 
It returns a new tensor with reduced dimensions if no specific `dim` is provided, or modifies the input tensor along the specified `dim`.

**Implementation Details:**

- Uses PyTorch's built-in tensor shape and dim methods.
- Ensures negative indices are adjusted positively.

**Dependencies/Notable Requirements:**

- PyTorch library ([torch](https://pytorch.org/))
- Meta device for tensor allocation


key: chroma:document
string_value:
CODE:

def torch_tensor_squeeze(self, dim=None):


SUMMARY:
**Summary**

`torchTensorSqueezeMethod`: A method that applies `torch_squeeze` to the tensor in the same context (`self`). 

**Key Details:**

* **Method Signature:** `torch_tensor_squeeze(self, dim=None)`
* **Implementation:** Delegates to `torch_squeeze(self, dim)`, implying a dependency on an external `torch_squeeze` function.


key: chroma:document
string_value:
CODE:

def torch_unsqueeze(input, dim):
    shape = list(input.shape)
    if dim < 0:
        dim = input.dim() + 1 + dim
    shape.insert(dim, 1)


SUMMARY:
**Summary**

`torch_unsqueeze`: Function that inserts a singleton dimension at the specified position (`dim`) in a tensor, ensuring compatibility with PyTorch's dynamic reshaping.

**Key Implementation Details:**

* Inserts a singleton dimension using `input.dim() + 1 + dim` as the new dimension index.
* Generates an empty tensor with the updated shape and device "meta".

**Notable Dependencies:**

* `torch`: The primary dependency, providing tensor operations and device management.


key: chroma:document
string_value:
CODE:

def torch_tensor_unsqueeze(self, dim):


SUMMARY:
**Summary:**

Re-implementation of `torch_unsqueeze` function with variable dimension handling. The original implementation is wrapped to maintain consistency.

* Purpose: Enhance the functionality of `torch_unsqueeze` by allowing dynamic dimension specification.
* Notable details:
	+ Original implementation handled only two dimensions (0 and 1).
	+ Re-implementation allows for any specified dimension (`dim` parameter).
* Dependencies:
	+ Utilizes `torch_unsqueeze` from PyTorch library (_PyTorch is not used within this code snippet, however it will be utilized in the underlying Torch package.)
		- torch Unsqueeze
* Design choice:
	+ The function merely forwards the call to the original `_torch_unsqueeze` implementation.
	+ Function's effectiveness heavily relies on parent class/parent module's `_torch_unsqueeze` handling.


key: chroma:document
string_value:
CODE:

def torch_unique_consecutive(input, **kwargs):
    output = torch.unique_consecutive(torch.zeros_like(input, device="cpu"), **kwargs)
    if isinstance(output, torch.Tensor):
        return output.to("meta")
    else:


SUMMARY:
**Summary**

Returns the unique consecutive elements of an input tensor while running on the "meta" device.

### Key Functionality

* Calculates unique consecutive elements using `torch.unique_consecutive`
* Returns output as either a single tensor or a tuple of tensors, depending on original input type
* Moves output to "meta" device ("cpu" in this implementation)

### Implementation Details

* Uses `torch.zeros_like` to create a dummy input tensor with the same shape and data type as the input (`device="cpu"` parameter)
* Returns converted output using `map` function, to move it from the original device to "meta"

### Dependencies/Requirements

* torch.unique_consecutive
* torch.zeros_like (with device set to "cpu")
* met devices (presumably from a PyTorch backend)

### Interfaces/Inheritance

* Uses torch.Tensor interface for output


key: chroma:document
string_value:
CODE:

def torch_nn_functional_one_hot(tensor, num_classes=-1):
    if num_classes < 0:
        raise ValueError("Don't support automatic num_classes inference for MetaTensor analysis")
    shape = list(tensor.shape) + [num_classes]


SUMMARY:
**torch_nn_functional_one_hot Function Summary**

* **Purpose**: Creates an empty Metatensor with shape increased by one dimension for each class, suitable for one-hot encoding.
* **Key Implementation Details**:
	+ Uses NumPy's tensor replication to set `shape += [num_classes]`.
	+ Returns a PyTorch Tensor on the "meta" device.
	+ Raises ValueError if `num_classes` is invalid (i.e., < 0).
* **Notable Dependencies and Requirements**: None.
* **Interface Signature**: 
```
torch_nn_functional_one_hot(tensor: torch.Tensor, num_classes: int = -1) -> torch.Tensor
```


key: chroma:document
string_value:
CODE:

def torch_nn_functional_scaled_dot_product_attention(
    query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None
):
    target_length = query.shape[-2]
    head_dim = value.shape[-1]


SUMMARY:
**Summary:**
`torch_nn_functional_scaled_dot_product_attention`

*   Purpose:
    Creates an attention function for scaled dot product attention in PyTorch Neural Networks. It allocates memory for the attention weights and returns a tensor with shape `(batch_size, query_length, head_dim)`.
*   Implementation details:
    *   Handles attention masking via `attn_mask` parameter.
    *   Supports dropout via `dropout_p` parameter.
    *   Allows causal attention (`is_causal=True`) by allocating fewer dimensions for the target length.
    *   Optionally scales the attention weights via `scale=None`.
*   Dependencies:
    *   Uses `torch.empty()` to allocate memory, with shape `(batch_size, query_length, head_dim)`.
    *   Requires `query` and `value` tensors as inputs.
    
**Key method signature:**

`torch.nn.functional.scaled_dot_product_attention(query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None)`


key: chroma:document
string_value:
CODE:

def torch_nn_mseloss(self, input, target):
    if self.reduction == "none":
        shape = target.shape
    else:
        shape = (1,)


SUMMARY:
**torch_nn_mseloss Function Summary**

The `torch_nn_mseloss` function calculates the Mean Squared Error (MSE) loss under PyTorch Neural Networks.

**Key Implementation Details:**

* Compares to target output using MSE calculation
* Uses `torch.empty` to create an output tensor with shape determined by `self.reduction`

**Method Signature:** `torch.nn.MSELoss()(input, target"`

**Interface Inheritance:** `torch.nn.Module`


key: chroma:document
string_value:
CODE:

def torch_nn_crossentropyloss(self, input, target):
    if self.reduction == "none":
        shape = target.shape
    else:
        shape = (1,)


SUMMARY:
**Concise Summary:**

`torch_nn_crossentropyloss`: Computes the non-reduced neural network cross-entropy loss.

**Key Details:**

* Takes in `input` and `target` tensors as inputs.
* Returns an empty tensor with shape `(1,)`, determined by whether a reduction is applied (`self.reduction`) or not.
* Output device set to `"meta"`.


key: chroma:document
string_value:
CODE:

def torch_nn_bcewithlogitsloss(self, input, target):
    if self.reduction == "none":
        shape = target.shape
    else:
        shape = (1,)


SUMMARY:
**Summary:**

The `torch_nn_bcewithlogitsloss` function computes the binary cross-entropy loss with logit transformation using PyTorch. It takes in `input` and `target` tensors as arguments.

**Key Functionality:**

* Computes binary cross-entropy loss between `input` and `target` tensors, transformed by logit function.
* Applies specified reduction mode ("none" or other), affecting output shape.
* Returns a new tensor with the computed loss, filled in memory instead of on GPU (due to "meta" device).

**Implementation Details:**

* No specific algorithms or patterns are applied; basic loss computation is performed.

Note: I've assumed the `torch` library is being used, as indicated by the `torch.empty` function call.


key: chroma:document
string_value:
CODE:

def operator_getitem(a, b):
    def to_concrete(t):
        if isinstance(t, torch.Tensor):
            concrete = torch.ones_like(t, device="cpu")
            if concrete.dtype in [torch.float16, torch.float32, torch.float64, torch.int32]:
                concrete = concrete.to(torch.int64)
            return concrete
        return t

    if isinstance(a, torch.Tensor):
        # TODO: infer shape without performing the computation.
        if isinstance(b, tuple):
            b = tuple(map(to_concrete, b))
        else:
            b = to_concrete(b)
        return operator.getitem(torch.empty_like(a, device="cpu"), b).to("meta")


SUMMARY:
**Function Summary: `operator_getitem`**

The `operator_getitem` function appears to be a custom implementation of the container data structure's indexing operation. It is used to access elements in a tensor-like object.

**Primary Functionality:** Indexing operation for tensor-like objects, handling conversions and type casting where necessary.

**Key Implementation Details:**

* Uses the `torch` library to perform operations.
* Conversions are performed using the `to_concrete` function:
	+ Converts `torch.Tensor` instances to corresponding integer types (signed 64-bit).
	+ Handles tuple indexing by applying `to_concrete` to each member element.
* Utilizes an empty tensor as a placeholder during indexing, leveraging `operator.getitem`.

**Dependences and Requirements:**

* `torch`: PyTorch library for GPU acceleration.
* `operator`: Built-in Python module providing the `getitem` function.


key: chroma:document
string_value:
CODE:

class HFProxy(Proxy):
    """
    Proxy that uses metadata to handle data-dependent control-flow.
    """

    def install_metadata(self, metadata):
        self._metadata = metadata

    @property
    def shape(self):
        return self.tracer.create_proxy("call_method", "size", (self,), {})

    @property
    def device(self):
        # Hack so we can track when devices are used. During meta-tensor propagation,
        # replace these values with a constant 'meta'
        return MetaDeviceAttribute(self, "device")

    def __len__(self):
        if hasattr(self, "_metadata") and self._metadata is not None:
            return len(self._metadata)
        return super().__len__()

    def __bool__(self):
        if hasattr(self, "_metadata") and self._metadata is not None:
            return self._metadata
        return super().__bool__()

    def __getattr__(self, k):
        if k == "_metadata":
            return self.__getattribute__(k)
        # note: not added to the graph yet, if this is a method call
        # we peephole optimize to the method invocation
        return HFAttribute(self, k)

    def __setitem__(self, indices, values):
        return self.tracer.create_proxy("call_function", operator.setitem, (self, indices, values), {})

    def __contains__(self, key):
        if hasattr(self, "_metadata") and self._metadata is not None:
            return key in self._metadata


SUMMARY:
**Class Summary: HFProxy**

**Purpose/Functionality:**
The `HFProxy` class serves as a metadata-driven proxy for handling data-dependent control flow. It provides a way to manipulate and track metadata associated with the object.

**Key Implementation Details:**

* Uses `Metadata` to store and manage data.
* Implements getter and setter methods for accessing and modifying `_metadata`.
* Replaces `device` attribute with `MetaDeviceAttribute` during meta-tensor propagation.
* Overrides `__len__`, `__bool__`, `__getattr__`, `__setitem__`, and `__contains__` methods to utilize metadata.

**Dependencies/Requirements:**

* `tracer`: an object used for creating proxy calls.
* `MetaDeviceAttribute`: a specialized attribute class.
* `HFAttribute`: a helper class for attribute access.
* `Metadata`: a data structure storing metadata.

Note: External dependencies may include `MetaTensor` and `OP` in the context of tensor processing.


key: chroma:document
string_value:
CODE:

class HFAttribute(HFProxy):
    def __init__(self, root, attr: str):
        self.root = root
        self.attr = attr
        self.tracer = root.tracer
        self._node = None

        if hasattr(self.root, "_metadata"):
            self.install_metadata(getattr(self.root._metadata, attr))

    @property
    def node(self):
        # the node for attributes is added lazily, since most will just be method calls
        # which do not rely on the getitem call
        if self._node is None:
            self._node = self.tracer.create_proxy("call_function", builtins.getattr, (self.root, self.attr), {}).node
        return self._node

    def __call__(self, *args, **kwargs):


SUMMARY:
**Summary: HFAttribute Class**

The `HFAttribute` class extends `HFProxy` and provides lazy access to attributes of its underlying root object.

**Primary Purpose:** Provides read-only access to attributes of the root object with caching and tracing capabilities.

**Key Implementation Details:**

- Employs a property 'node' that lazily retrieves and stores the actual attribute value using the `_tracer.create_proxy` method.
- Immutably implements `__call__` for executing the root object's attribute as a method call, utilizing the same proxy mechanism.

**Notable Dependencies or Requirements:**

- Utilizes `HFProxy` class
- Relies on `builtins.getattr` function
- Inherently requires access to the 'tracer' attribute of its root object.


key: chroma:document
string_value:
CODE:

class MetaDeviceAttribute(HFAttribute):


SUMMARY:
Here is a concise summary of the provided class code chunk:

**Class Summary:**

`MetaDeviceAttribute`
========================

A subclass of `HFAttribute`, defining an attribute metadata for devices. The class inherits basic functionality from its parent class without providing any additional implementation.

**Key Implementation Details:**

* Inherited from `HFAttribute`

**Notable Dependencies:**

* `HFAttribute`: parent class definition

This summary focuses on the main purpose and key design choices of the class, omitting unnecessary details while still conveying the necessary context.


key: chroma:document
string_value:
CODE:

class HFCacheProxy(HFProxy):
    """
    Proxy that represents an instance of `transformers.cache_utils.Cache`.
    """

    def install_orig_cache_cls(self, orig_cache_cls: Type[Cache]):
        self._orig_cache_cls = orig_cache_cls

    @property
    def __class__(self):
        if not hasattr(self, "_orig_cache_cls"):
            raise RuntimeError("The original Cache class must be installed to the HFCacheProxy.")


SUMMARY:
**Class Summary: `HFCacheProxy`**

* **Purpose**: A cache proxy that wraps an instance of `transformers.cache_utils.Cache`, enabling transformation of the original class.

### Key Implementation Details

* The `install_orig_cache_cls` method sets `_orig_cache_cls` with the provided `orig_cache_cls`.
* The `_class` property returns the patched class from `tracer._CLASSES_TO_PATCH[_orig_cache_cls]`.

### Notable Dependencies and Requirements

* `transformers.cache_utils.Cache`: A required parent class.
* `HFProxy`: The base class for this proxy.


key: chroma:document
string_value:
CODE:

def create_wrapper(
    function: Callable,
    op_type: Union[Literal["call_function"], Literal["call_method"], Literal["get_attr"]],
    proxy_factory_fn: Optional[Callable[[Node], Proxy]] = None,
) -> Callable:
    @functools.wraps(function)
    def wrapper(*args, **kwargs):
        if not is_fx_tracing():
            return function(*args, **kwargs)

        found_proxies = []

        def check_proxy(a):
            if isinstance(a, Proxy):
                found_proxies.append(a)

        torch.fx.node.map_aggregate(args, check_proxy)
        torch.fx.node.map_aggregate(kwargs, check_proxy)

        if len(found_proxies) > 0:
            tracer = found_proxies[0].tracer
            if op_type == "call_function":
                target = function
            elif op_type == "call_method":
                target = function.__name__
            elif op_type == "get_attr":
                target = function.__name__
            else:
                raise ValueError(f"op_type {op_type} not supported.")
            return tracer.create_proxy(op_type, target, args, kwargs, proxy_factory_fn=proxy_factory_fn)
        else:
            return function(*args, **kwargs)



SUMMARY:
**Function Summary: create_wrapper**

 Creates a wrapper function that conditionally wraps the input `function` with tracing functionality using TensorBoardX's `torch.fx` module.

**Primary Purpose:** Conditionally wraps a function with tracing and proxying functionality based on the `op_type` parameter.

**Key Implementation Details:**

* Uses `functools.wraps` to preserve original function metadata.
* Checks if tracing is enabled and calls the `function` directly if not.
* Utilizes `torch.fx.node.map_aggregate` to find proxies in arguments and keyword arguments.
* If proxy(s) are found, creates a target based on `op_type` and returns a new probe with the proxy factory function.

**Notable Dependencies:**

* `torch`: Core library for TensorBoardX's tracing functionality.
* `functools`: Used for preserving original function metadata.
* `torch.fx`: Module providing tracing and proxying functionality.


key: chroma:document
string_value:
CODE:

class HFProxyableClassMeta(type):
    """
    Metaclass that creates a class with its main methods wrapped to be proxyable.
    """

    def __new__(
        cls,
        name: str,
        bases: Tuple[Type, ...],
        attrs: Dict[str, Any],
        proxy_factory_fn: Optional[Callable[[Node], Proxy]] = None,
    ):
        cls = super().__new__(cls, name, bases, attrs)
        for attr_name in dir(cls):
            attr = getattr(cls, attr_name, None)
            if attr is None:
                continue
            if attr_name == "__init__":
                op_type = "call_function"
            elif attr_name.startswith("__"):
                op_type = None
            elif inspect.ismethod(attr):
                op_type = "call_function"
            elif inspect.isfunction(attr):
                op_type = "call_method"
            else:
                op_type = None
            if op_type is not None:
                setattr(cls, attr_name, create_wrapper(attr, op_type, proxy_factory_fn=proxy_factory_fn))


SUMMARY:
**HFProxyableClassMeta Metaclass Summary**

### Purpose and Functionality

Creates a class with its main methods wrapped to be proxyable. This metaclass allows for dynamic wrapping of class attributes as proxyable methods, enabling modularization and extension of existing classes.

### Key Implementation Details

* Wraps instance methods (`__init__`, `create_wrapper`) with the "call_function" operation type.
* Tunes method call behavior based on attribute type (method, function, or other).
* Utilizes a wrapper creation function (`create_wrapper`) to generate proxyable class attributes.

### Notable Dependencies and Requirements

* `inspect`: for checking if an attribute is a method or function.
* `create_wrapper`: an external function not shown in this code snippet.
* `proxy_factory_fn`: an optional dependency (Callable[[Node], Proxy]) for customizing proxy creation behavior.

This metaclass assumes the presence of the `create_wrapper` function, which is not included here. Its implementation would further enhance the proxyable wrapping functionality.


key: chroma:document
string_value:
CODE:

def gen_constructor_wrapper(target: Callable) -> Tuple[Callable, Callable]:
    """
    Wraps `target` to be proxyable. Used for tensor creators like `torch.ones`, `torch.arange` and so on.
    """
    wrapper = create_wrapper(target, "call_function")


SUMMARY:
**Function Summary**

```python
gen_constructor_wrapper:
    Purpose: Wraps a target function to enable proxy callable execution.

    **Key Details:**
    - Uses `create_wrapper` function to generate the wrapped call function.
    - Returns both the wrapped and original callable functions.

    **Implementation Dependencies:**
    - External dependency on `create_wrapper` function from torch library.
```

This summary extracts the main idea of the function, highlights key implementation details, and mentions any notable external dependencies. It also preserves the exact class/function name (`Callable`, `Tuple`) and precise technical term (`Proxyable`).


key: chroma:document
string_value:
CODE:

def _proxies_to_metas(v):
    """Returns the underlying metadata for HFProxies, and behaves like the identity for the others."""
    if isinstance(v, MetaDeviceAttribute):
        return "meta"
    if isinstance(v, torch.fx.Proxy):
        if not (isinstance(v, HFProxy) and hasattr(v, "_metadata")):
            raise RuntimeError(f"No metadata was found for {v}")
        return v._metadata


SUMMARY:
**Summary**

The `_proxies_to_metas` function returns the underlying metadata for an object. It is designed to behave as the identity for objects of type `MetaDeviceAttribute`, but provides metadata for objects of type `torch.fx.Proxy`. If the metadata is not found, it raises a `RuntimeError`.

**Key Points**

* Primary purpose: Returns metadata for HFProxies and other objects.
* Key implementation detail: Checks if an object has `_metadata` attribute to determine its class.
* Notable dependencies:
	+ `MetaDeviceAttribute`
	+ `torch.fx.Proxy`
	+ `HFProxy`

Note that the function signature is `_proxies_to_metas(v)`, where `v` is the input object being analyzed.


key: chroma:document
string_value:
CODE:

def create_cache_proxy_factory_fn(orig_cache_cls: Type[Cache]) -> Callable[[Node], HFCacheProxy]:
    def cache_proxy_factory_fn(n: Node) -> HFCacheProxy:
        global _CURRENT_TRACER
        if not isinstance(_CURRENT_TRACER, HFTracer):
            raise RuntimeError("Cannot create HFCacheProxy because there is no HFTracer currently tracing.")
        cache_proxy = HFCacheProxy(n, _CURRENT_TRACER)
        cache_proxy.install_orig_cache_cls(orig_cache_cls)
        return cache_proxy



SUMMARY:
**Function Summary: Create Cache Proxy Factory Function**

### Purpose
Create a factory function to generate an instance of `HFCacheProxy`, which wraps and configures the original cache class specified by `orig_cache_cls`.

### Implementation Details

*   Creates a `HFCacheProxy` instance with a given `Node` object and `_CURRENT_TRACER` tracer.
*   Installs and configures the original cache class using `orig_cache_cls`.
*   Returns an instance of `HFCacheProxy`.

#### Key Method Signatures:

*   `cache_proxy_factory_fn(n: Node) -> HFCacheProxy`: factory function to create a `HFCacheProxy` instance.

#### Notable Dependencies and Requirements:

*   `_CURRENT_TRACER` (object): an instance of `HFTracer`.
*   `orig_cache_cls` (class): the original cache class type.
*   `HFTracer`, `HFCacheProxy`, `Node`: related classes with interfaces.

### Primary Purpose
This function generates a factory function that creates and configures an instance of `HFCacheProxy`, enabling the use of the original cache class specified by `orig_cache_cls` within the context of a tracing session managed by `_CURRENT_TRACER`.


key: chroma:document
string_value:
CODE:

def _generate_random_int(low: int = 10, high: int = 20, forbidden_values: Optional[List[int]] = None):
    if forbidden_values is None:
        forbidden_values = []
    value = random.randint(low, high)
    while value in forbidden_values:
        value = random.randint(low, high)


SUMMARY:
**Summary: `generate_random_int` Function**

* **Purpose**: Generates a random integer within a specified range while excluding forbidden values.
* **Functionality**: Uses `random.randint` to generate a unique value until it is not in the provided `forbidden_values` list.

**Key Implementation Details:**
- Uses `Optional[List[int]]` as the type of `forbidden_values`
- Calls `random.randint(low, high)` repeatedly until a valid value is generated

**Notable Dependencies/Requirements:**

* `random` module (imported implicitly)

Note that this function does not inherit or implement any interfaces. It relies on the default methods provided by Python's built-in `random` module.


key: chroma:document
string_value:
CODE:

class HFTracer(Tracer):
    """
    Tracer that is able to symbolically trace models from the library. To do that, it uses the HFProxy instead of the
    regular PyTorch torch.fx.Proxy.
    """

    # Feature flag for proxying accesses to buffer values
    proxy_buffer_attributes: bool = True
    allow_insert_stateless_mods: bool = True
    _TORCH_METHODS_TO_PATCH = [
        "arange",
        "zeros",
        "ones",
        "full",
        "full_like",
        "eye",
        "empty",
        "tensor",
        "clamp",
        "finfo",
        "tril",
    ]
    _CLASSES_TO_PATCH = {
        Cache: ProxyableCache,
        DynamicCache: ProxyableDynamicCache,
        SinkCache: ProxyableSinkCache,
        StaticCache: ProxyableStaticCache,
    }

    supported_archs = (PreTrainedModel,) if not is_peft_available() else (PreTrainedModel, PeftModel)

    def __init__(self, autowrap_modules=(math,), autowrap_functions=()):
        super().__init__(autowrap_modules=autowrap_modules, autowrap_functions=autowrap_functions)

        if not is_torch_fx_available():
            raise ImportError(
                f"Found an incompatible version of torch. Found version {get_torch_version()}, but only version "
                f"{TORCH_FX_REQUIRED_VERSION} is supported."
            )

    def _generate_dummy_input(
        self, model: "PreTrainedModel", input_name: str, shape: List[int], input_names: List[str]
    ) -> Dict[str, torch.Tensor]:
        """Generates dummy input for model inference recording."""
        # Retrieving the model class, either from the "class_for_deserialization" attribute if the model was restored
        # from pickle, or from the "__class__" attribute in the general case.
        model_class_name = getattr(model, "class_for_deserialization", model.__class__).__name__
        device = model.device
        inputs_dict = {}

        # when tracing a model with KV cache, we simply need to unsure that the KV cache length is larger than one to
        # rightfully pass certain controlflows (Example: https://github.com/huggingface/transformers/blob/5c8d941d66734811d2ef6f57f15b44f7fb7a98c4/src/transformers/modeling_attn_mask_utils.py#L162).
        # After tracing, the model can then still be used with arbitrary lengths different than the one used during tracing.
        kv_cache_length = 5

        if input_name in ["labels", "start_positions", "end_positions"]:
            batch_size = shape[0]
            if model_class_name in [
                *get_values(MODEL_FOR_NEXT_SENTENCE_PREDICTION_MAPPING_NAMES),
                *get_values(MODEL_FOR_MULTIPLE_CHOICE_MAPPING_NAMES),
                *get_values(MODEL_FOR_IMAGE_CLASSIFICATION_MAPPING_NAMES),
                *get_values(MODEL_FOR_BACKBONE_MAPPING_NAMES),
                *get_values(MODEL_FOR_AUDIO_CLASSIFICATION_MAPPING_NAMES),
            ]:
                inputs_dict["labels"] = torch.zeros(batch_size, dtype=torch.long, device=device)
            elif model_class_name in [
                *get_values(MODEL_FOR_QUESTION_ANSWERING_MAPPING_NAMES),
                *get_values(MODEL_FOR_DOCUMENT_QUESTION_ANSWERING_MAPPING_NAMES),
                "XLNetForQuestionAnswering",
            ]:
                inputs_dict["start_positions"] = torch.zeros(batch_size, dtype=torch.long, device=device)
                inputs_dict["end_positions"] = torch.zeros(batch_size, dtype=torch.long, device=device)
            elif model_class_name in get_values(MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING_NAMES):
                if not hasattr(model.config, "problem_type") or model.config.problem_type is None:
                    raise ValueError(
                        "Could not retrieve the problem type for the sequence classification task, please set "
                        'model.config.problem_type to one of the following values: "regression", '
                        '"single_label_classification", or "multi_label_classification".'
                    )

                if model.config.problem_type == "regression":
                    labels_shape = (batch_size, model.config.num_labels)
                    labels_dtype = torch.float32
                elif model.config.problem_type == "single_label_classification":
                    labels_shape = (batch_size,)
                    labels_dtype = torch.long
                elif model.config.problem_type == "multi_label_classification":
                    labels_shape = (batch_size, model.config.num_labels)
                    labels_dtype = torch.float32
                else:
                    raise ValueError(
                        'Expected model.config.problem_type to be either: "regression", "single_label_classification"'
                        f', or "multi_label_classification", but "{model.config.problem_type}" was provided.'
                    )
                inputs_dict["labels"] = torch.zeros(*labels_shape, dtype=labels_dtype, device=device)

            elif model_class_name in [
                *get_values(MODEL_FOR_PRETRAINING_MAPPING_NAMES),
                *get_values(MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMES),
                *get_values(MODEL_FOR_CAUSAL_LM_MAPPING_NAMES),
                *get_values(MODEL_FOR_MASKED_LM_MAPPING_NAMES),
                *get_values(MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES),
                *get_values(MODEL_FOR_SEMANTIC_SEGMENTATION_MAPPING_NAMES),
                "GPT2DoubleHeadsModel",
                "PeftModelForCausalLM",
                "PeftModelForSeq2SeqLM",
            ]:
                inputs_dict["labels"] = torch.zeros(shape, dtype=torch.long, device=device)
            elif model_class_name in [*get_values(MODEL_FOR_CTC_MAPPING_NAMES)]:
                inputs_dict["labels"] = torch.zeros(shape, dtype=torch.float32, device=device)
            else:
                raise NotImplementedError(
                    f"Generating the dummy input named {input_name} for {model_class_name} is not supported yet."
                )
        elif "pixel_values" in input_name:
            batch_size = shape[0]
            image_size = getattr(model.config, "image_size", None)
            if image_size is None:
                if hasattr(model.config, "vision_config"):
                    image_size = model.config.vision_config.image_size
                elif hasattr(model.config, "encoder"):
                    image_size = model.config.encoder.image_size
                else:
                    image_size = (_generate_random_int(), _generate_random_int())

            # If no num_channels is in the config, use some arbitrary value.
            num_channels = getattr(model.config, "num_channels", 3)
            if not isinstance(image_size, collections.abc.Iterable):
                image_size = (image_size, image_size)
            height, width = image_size
            inputs_dict[input_name] = torch.zeros(
                batch_size, num_channels, height, width, dtype=torch.float32, device=device
            )
        elif "bbox" in input_name:
            inputs_dict[input_name] = torch.zeros(*shape, 4, dtype=torch.float, device=device)
        elif "input_features" in input_name:
            inputs_dict[input_name] = torch.zeros(
                *shape, model.config.input_feat_per_channel, dtype=torch.float, device=device
            )
        elif "inputs_embeds" in input_name:
            batch_size = shape[0]

            if (
                getattr(model.config, "embedding_size", None) is not None
                and model.config.model_type != "megatron-bert"
            ):
                embedding_size = model.config.embedding_size
            else:
                embedding_size = model.config.hidden_size

            if len(shape) == 3:
                # (batch_size, num_choices, sequence_length, embedding_size)
                embedding_shape = (batch_size, shape[1], shape[2], embedding_size)
            else:
                # (batch_size, sequence_length, embedding_size)
                embedding_shape = (batch_size, shape[1], embedding_size)

            inputs_dict[input_name] = torch.zeros(embedding_shape, dtype=torch.float, device=device)
        elif "visual_feats" in input_name:
            inputs_dict[input_name] = torch.zeros(
                shape
                + [
                    model.config.visual_feat_dim,
                ],
                dtype=torch.float,
                device=device,
            )
        elif "visual_pos" in input_name:
            inputs_dict[input_name] = torch.zeros(
                shape
                + [
                    model.config.visual_pos_dim,
                ],
                dtype=torch.float,
                device=device,
            )
        elif "inputs" in input_name:
            inputs_dict[input_name] = torch.zeros(*shape, dtype=torch.float, device=device)
        elif "input_values" in input_name:
            batch_size, _ = shape
            # Generating big sequence length for audio inputs.
            seq_length = _generate_random_int(low=10000, high=20000)
            inputs_dict[input_name] = torch.zeros(batch_size, seq_length, dtype=torch.float, device=device)
        elif "mask" in input_name:
            if "past_key_values" in input_names:
                mask_shape = [shape[0], shape[1] + kv_cache_length]
            else:
                mask_shape = shape

            inputs_dict[input_name] = torch.zeros(mask_shape, dtype=torch.long, device=device)
        elif "ids" in input_name:
            inputs_dict[input_name] = torch.zeros(shape, dtype=torch.long, device=device)
        elif "past_key_values" in input_name:
            if model.config.model_type not in _FX_SUPPORTED_MODELS_WITH_KV_CACHE:
                raise NotImplementedError(
                    f"Symbolic trace with past_key_values input is not supported yet for the model {model.config.model_type}. Please open an issue or a PR in Transformers repository if you would like to see the support added."
                )
            num_heads = model.config.num_attention_heads
            head_dim = model.config.hidden_size // model.config.num_attention_heads

            cache_shape = (shape[0], num_heads, kv_cache_length, head_dim)
            pkv = tuple(
                (
                    torch.rand(cache_shape, dtype=torch.float, device=device),
                    torch.rand(cache_shape, dtype=torch.float, device=device),
                )
                for i in range(model.config.num_hidden_layers)
            )
            inputs_dict[input_name] = pkv
        else:
            shape_with_hidden_size = shape + [model.config.hidden_size]
            inputs_dict[input_name] = torch.zeros(shape_with_hidden_size, dtype=torch.float, device=device)

        return inputs_dict

    def create_proxy(self, kind, target, args, kwargs, name=None, type_expr=None, proxy_factory_fn=None):
        rv = super().create_proxy(kind, target, args, kwargs, name, type_expr, proxy_factory_fn)

        if kind == "placeholder" and target in self.meta_args:
            rv.install_metadata(self.meta_args[target])
            return rv

        if target in self.orig_fns:
            # NOTE: tensor constructors in PyTorch define the `device` argument as
            # *kwargs-only*. That is why this works. If you add methods to
            # _TORCH_METHODS_TO_PATCH that do not define `device` as kwarg-only,
            # this will break and you will likely see issues where we cannot infer
            # the size of the output.
            if "device" in kwargs:
                kwargs["device"] = "meta"

        try:
            args_metas = torch.fx.node.map_aggregate(args, _proxies_to_metas)
            kwargs_metas = torch.fx.node.map_aggregate(kwargs, _proxies_to_metas)

            should_install_metadata = True

            self._disable_module_getattr = True
            self._disable_call_module = True

            if kind == "call_function":
                meta_target = _MANUAL_META_OVERRIDES.get(target, target)
                meta_out = meta_target(*args_metas, **kwargs_metas)
                if isinstance(meta_out, torch.Tensor):
                    meta_out = meta_out.to(device="meta")
            elif kind == "call_method":
                method = getattr(args_metas[0].__class__, target)
                meta_target = _MANUAL_META_OVERRIDES.get(method, method)
                meta_out = meta_target(*args_metas, **kwargs_metas)
            elif kind == "call_module":
                if not hasattr(self, "orig_forward"):
                    raise AttributeError(f"{self} does not have an attribute called orig_forward")
                mod = self.root.get_submodule(target)
                mod_type = type(mod)
                if mod_type in _MANUAL_META_OVERRIDES:
                    meta_out = _MANUAL_META_OVERRIDES[mod_type](mod, *args_metas, **kwargs_metas)
                else:
                    meta_out = self.orig_forward(*args_metas, **kwargs_metas)
            elif kind == "get_attr":
                attr_itr = self.root
                atoms = target.split(".")
                for atom in atoms:
                    attr_itr = getattr(attr_itr, atom)
                if isinstance(attr_itr, torch.Tensor):
                    meta_out = attr_itr.to(device="meta")
                else:
                    meta_out = attr_itr
            else:
                should_install_metadata = False

            if should_install_metadata:
                if not isinstance(rv, Proxy):
                    raise ValueError("Don't support composite output yet")
                rv.install_metadata(meta_out)

        except Exception as e:
            if _IS_IN_DEBUG_MODE:
                warnings.warn(f"Could not compute metadata for {kind} target {target}: {e}")

        self._disable_module_getattr = False
        self._disable_call_module = False

        return rv

    # Replaced by .getattr from PyTorch 1.13
    def _module_getattr(self, attr, attr_val, parameter_proxy_cache):
        if getattr(self, "_disable_module_getattr", False):
            return attr_val
        else:

            def maybe_get_proxy_for_attr(attr_val, collection_to_search, parameter_proxy_cache):
                for n, p in collection_to_search:
                    if attr_val is p:
                        if n not in parameter_proxy_cache:
                            kwargs = {}
                            if "proxy_factory_fn" in inspect.signature(self.create_proxy).parameters:
                                kwargs["proxy_factory_fn"] = (
                                    None
                                    if not self.param_shapes_constant
                                    else lambda node: ParameterProxy(self, node, n, attr_val)
                                )
                            val_proxy = self.create_proxy("get_attr", n, (), {}, **kwargs)  # type: ignore[arg-type]
                            parameter_proxy_cache[n] = val_proxy
                        return parameter_proxy_cache[n]
                return None

            if isinstance(attr_val, torch.nn.Parameter):
                maybe_parameter_proxy = maybe_get_proxy_for_attr(
                    attr_val, self.root.named_parameters(), parameter_proxy_cache
                )
                if maybe_parameter_proxy is not None:
                    return maybe_parameter_proxy

            if self.proxy_buffer_attributes and isinstance(attr_val, torch.Tensor):
                maybe_buffer_proxy = maybe_get_proxy_for_attr(
                    attr_val, self.root.named_buffers(), parameter_proxy_cache
                )
                if maybe_buffer_proxy is not None:
                    return maybe_buffer_proxy

            return attr_val

    # Needed for PyTorch 1.13+
    def getattr(self, attr: str, attr_val: Any, parameter_proxy_cache: Dict[str, Any]):
        return self._module_getattr(attr, attr_val, parameter_proxy_cache)

    def call_module(self, m, forward, args, kwargs):
        if getattr(self, "_disable_call_module", False):
            return forward(*args, **kwargs)
        self.orig_forward = forward
        return super().call_module(m, forward, args, kwargs)

    def proxy(self, node):
        return HFProxy(node, self)

    @contextlib.contextmanager
    def patch_for_tracing(self, root: Union[torch.nn.Module, Callable[..., Any]]):
        # Patching torch functions
        self.patched_torch_methods = {
            target: gen_constructor_wrapper(getattr(torch, target)) for target in self._TORCH_METHODS_TO_PATCH
        }
        self.orig_fns = set()

        for name, (wrapper, orig) in self.patched_torch_methods.items():
            setattr(torch, name, wrapper)
            self.orig_fns.add(orig)

        # Patching classes
        patched = []
        module_of_model = inspect.getmodule(root)
        for name, mod in sys.modules.items():
            if module_of_model is not None and mod is not module_of_model:
                continue
            if not name.startswith("transformers"):
                continue
            for orig_cls, patched_cls in self._CLASSES_TO_PATCH.items():
                for attr_name, attr in mod.__dict__.items():
                    if attr is orig_cls:
                        patched.append((mod, attr_name, orig_cls))
                        setattr(mod, attr_name, patched_cls)

        yield

        # Restoring patched functions and classes.
        for name, (_, orig) in self.patched_torch_methods.items():
            setattr(torch, name, orig)
        self.patched_torch_methods = {}
        self.orig_fns = set()

        for mod, attr_name, orig_cls in patched:
            setattr(mod, attr_name, orig_cls)

    def trace(
        self,
        root: Union[torch.nn.Module, Callable[..., Any]],
        concrete_args: Optional[Dict[str, Any]] = None,
        dummy_inputs: Optional[Dict[str, Any]] = None,
        complete_concrete_args_with_inputs_not_in_dummy_inputs: bool = True,
    ) -> Graph:
        """
        Traces `root` and returns the corresponding FX `torch.fx.Graph` representation. `root` can either be a
        `torch.nn.Module` instance or a Python callable. Note that after this call, `self.root` may be different from
        the `root` passed in here. For example, when a free function is passed to `trace()`, we will create a
        `torch.nn.Module` instance to use as the root and add embedded constants to.

        Args:
            root (`torch.nn.Module` or  `Callable`):
                Either a `torch.nn.Module`` or a function to be traced through. If root is not a
                [`~transformers.PreTrainedModel`], then `dummy_inputs` must be passed, otherwise tracing will fail.
            concrete_args (`Dict[str, Any], *optional*):
                Concrete arguments that should not be treated as Proxies
            dummy_inputs (`Dict[str, Any]`, *optional*):
                The dummy inputs needed to handle data-dependent control-flow if `root` is not a
                [`~transformers.PreTrainedModel`]. It can also be used when `root` is a
                [`~transformers.PreTrainedModel`] to specify custom dummy inputs for a subset or all the model inputs.
            complete_concrete_args_with_inputs_not_in_dummy_inputs (`bool`, *optional*, defaults to `True`):
                If `True`, and `dummy_inputs` is specified, every argument that `root` can take that is not in
                `dummy_inputs` and not in `concrete_args` will be added to `concrete_args`, otherwise does nothing.

        Returns:
            `torch.fx.Graph`:
                A FX `torch.fx.Graph` representing the semantics of the passed-in `root`.

        """
        sig = inspect.signature(root.forward if isinstance(root, torch.nn.Module) else root)

        if concrete_args is None:
            concrete_args = {}

        if dummy_inputs is not None and complete_concrete_args_with_inputs_not_in_dummy_inputs:
            for param in sig.parameters.values():
                if param.name in dummy_inputs:
                    continue
                if param.default is inspect.Parameter.empty:
                    raise ValueError(f"You need to specify a default value for the parameter {param.name}.")
            concrete_args.update(
                {
                    p.name: p.default
                    for p in sig.parameters.values()
                    if (p.name not in dummy_inputs and p.name not in concrete_args)
                }
            )

        input_names = sig.parameters.keys() - concrete_args.keys()

        # Creating a random input shape to generate dummy inputs.
        batch_size = _generate_random_int()
        sequence_length = _generate_random_int()
        shape = [batch_size, sequence_length]

        if root.__class__.__name__ in get_values(MODEL_FOR_MULTIPLE_CHOICE_MAPPING_NAMES):
            num_choices = _generate_random_int(low=2, high=5)
            shape.insert(1, num_choices)

        inputs = dict(dummy_inputs) if dummy_inputs is not None else {}
        for input_name in input_names:
            if input_name in inputs:
                continue
            # We enforce that root must either be a PreTrainedModel or deserialized from a serialized traced model to
            # be able to use HFTracer._generate_dummy_input.
            if isinstance(root, self.supported_archs) or type(root).__qualname__.startswith(
                ("_deserialize_graph_module", "_CodeOnlyModule")
            ):
                inputs.update(self._generate_dummy_input(root, input_name, shape, input_names=input_names))
            else:
                raise RuntimeError(
                    f"Could not generate input named {input_name} for because root is not a"
                    " transformers.PreTrainedModel."
                )

        def to_meta(value):
            if isinstance(value, torch.Tensor):
                return value.to("meta")
            return value

        concrete_metas = pytree.tree_map(to_meta, inputs)

        for param in sig.parameters.values():
            if param.kind == inspect.Parameter.VAR_KEYWORD and param.name not in input_names:
                concrete_metas[f"**{param.name}"] = {}
        self.meta_args = concrete_metas

        global _CURRENT_TRACER
        _CURRENT_TRACER = self
        with self.patch_for_tracing(root):
            try:
                self.graph = super().trace(root, concrete_args=concrete_args)
            finally:
                _CURRENT_TRACER = None

        # This is necessary because concrete args are added as input to the traced module since
        # https://github.com/pytorch/pytorch/pull/55888.
        for node in self.graph.nodes:
            if node.op == "placeholder":
                # Removing default values for inputs as the forward pass will fail with them.
                if node.target in input_names:
                    node.args = ()
                    # Without this, torch.jit.script fails because the inputs type is Optional[torch.Tensor].
                    # It cannot infer on the attributes and methods the input should have, and fails.
                    node.type = torch.Tensor
                # It is a concrete arg so it is not used and should be removed.
                else:
                    to_visit = [node]
                    to_delete = collections.OrderedDict()
                    while to_visit:
                        n = to_visit.pop(0)
                        to_delete[n] = None
                        to_visit += list(n.users.keys())

                    for user in reversed(to_delete.keys()):
                        self.graph.erase_node(user)

            # TODO: solves GraphModule creation.
            # Without this, return type annotation "Tuple" is causing code execution failure.
            if node.op == "output":
                node.type = None

        return self.graph

    def _stateless_mod_instanciation_depends_on_proxies(self, mod: nn.Module) -> bool:
        """
        Whether the module was instantiated with Proxies. If that is the case, such module cannot be a leaf module
        because its attributes are input-dependent.
        """
        return any(isinstance(attr, Proxy) for attr in mod.__dict__.values())

    def _insert_module_as_submodule(self, mod: nn.Module) -> str:
        """
        Helper method which tries to insert a module that was not declared as submodule.
        """
        # If one of the module attributes is a Proxy, it means that its instantiation is input-dependent.
        # It is not possible to insert such modules, those should be traced through.
        if self._stateless_mod_instanciation_depends_on_proxies(mod):
            return ""
        idx = 0
        mod_name = mod.__class__.__name__.lower()
        path = f"{mod_name}_{idx}"
        already_inserted = False
        while hasattr(self.root, path):
            if getattr(self.root, path) is mod:
                already_inserted = True
                break
            path = f"{mod_name}_{idx}"
            idx += 1

        # No need to add multiple instances of the same module.
        if not already_inserted:
            self.root.add_module(path, mod)
        return path

    def path_of_module(self, mod: nn.Module) -> str:
        """
        Helper method to find the qualified name of `mod` in the Module hierarchy of `root`. For example, if `root` has
        a submodule named `foo`, which has a submodule named `bar`, passing `bar` into this function will return the
        string "foo.bar".

        Args:
            mod (str): The `Module` to retrieve the qualified name for.
        """
        try:
            return super().path_of_module(mod)
        except NameError as e:
            if self.allow_insert_stateless_mods and len(list(mod.parameters())) == 0 and len(list(mod.buffers())) == 0:
                path = self._insert_module_as_submodule(mod)
                return path
            raise e

    def is_leaf_module(self, m: torch.nn.Module, module_qualified_name: str) -> bool:
        return (not self._stateless_mod_instanciation_depends_on_proxies(m)) and super().is_leaf_module(
            m, module_qualified_name
        )

    @compatibility(is_backward_compatible=True)
    def keys(self, obj: "Proxy") -> Any:
        """Called when a proxy object is has the keys() method called.
        This is what happens when ** is called on a proxy. This should return an iterator if ** is supposed to work in
        your custom tracer.
        """
        attribute = HFAttribute(obj, "keys")()
        if obj.node.target.startswith("**"):
            return attribute._metadata


SUMMARY:
This code appears to be part of the Python Tracer (PyTorch) library. It provides functionality for creating proxy objects that can interact with PyTorch tensors and other PyTorch objects during tracing.

The main classes are `Proxy` and its subclasses, which wrap instances of PyTorch tensors and other types. The `Node` class is used to represent a node in the execution graph of the traced module.

Here's an overview of the code:

**Proxy Classes**

*   `Proxy`: A base class for proxy objects that wraps a tensor or other PyTorch object.
*   `ProxyWithMetadata` and `ProxyWithKeywordArgsWithMetadata`: Similar classes to `Proxy`, but also provide metadata (additional information) associated with the wrapped instance.

**Tracing Functions**

*   `_stateless_mod_instanciation_depends_on_proxies`: A boolean method that checks whether a module's attributes can depend on inputs (i.e., their instantiation depends on input tensors).
*   `_insert_module_as_submodule`: Helper function to insert a module into the `root` tracer's hierarchy, handling cases where modules' attribute instantiation is dependent on inputs.
*   `keys`: A method that provides an iterator for getting keys from proxy objects when ** is invoked.

**Node and Tracing**

*   `Node`: Represents a node in the execution graph of the traced module.
*   `path_of_module`: Helper function to retrieve the qualified name of a module within the given tracer's hierarchy.
*   `is_leaf_module`: Method that checks whether a module has only leaf modules as submodules.

This code is likely used in combination with PyTorch tracing functions, such as `torch.jit.trace()` and `torch.jit.apply_default()`, to create an execution graph of a PyTorch model and then execute it on the runtime environment.


key: chroma:document
string_value:
CODE:

def get_concrete_args(model: nn.Module, input_names: List[str]):
    sig = inspect.signature(model.forward)

    if not (set(input_names) <= set(sig.parameters.keys())):
        formatted_input_names = input_names[0] if len(input_names) == 1 else ", ".join(input_names)
        formatted_allowed_input_names = ", ".join(sig.parameters.keys())
        raise ValueError(
            f"The model does not have input(s) named: {formatted_input_names}, expected a subset of the following:"
            f" {formatted_allowed_input_names}"
        )



SUMMARY:
**Function Summary: `get_concrete_args`**

*Purpose:* Retrieves default argument values for given input names of a PyTorch neural network model.

*Key Implementation Details:* This function leverages the `inspect.signature` module to inspect the model's forward method. It compares the provided input names with the expected set of parameter keys in the signature, raising an error if any mismatch is found. The function then returns a dictionary containing default values for parameters not included in the expected input names.

**Method Signatures and Interfaces:** 
The `forward` method of the neural network model (`nn.Module`) is used to get the signature of the model's inputs and outputs.

**Dependencies and Requirements:**
- PyTorch library (`nn.Module`)
- Python standard library for `inspect.signature`
- Python list comprehension using `List[str]`


key: chroma:document
string_value:
CODE:

def is_model_supported(model: "PreTrainedModel"):


SUMMARY:
**Summary**

Functionally checks if a given `model` of type `PreTrainedModel` is supported by the codebase.

### Implementational Details
- Verifies class name membership in `_SUPPORTED_MODELS` list for each instance.

### Key Aspects
- Returns boolean indicating model support.
- Utilized class introspection to access model's name (`__class__.__name__`).

### Dependencies and Requirements
- None explicit dependencies stated. 
- Assumed presence of `_SUPPORTED_MODELS` (a list of supported model classes) defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def check_if_model_is_supported(model: "PreTrainedModel"):
    if not is_model_supported(model):
        supported_model_names = ", ".join(_SUPPORTED_MODELS)
        raise NotImplementedError(
            f"Model {model.__class__.__name__} is not supported yet, supported models: {supported_model_names}"


SUMMARY:
**Code Summary**

`check_if_model_is_supported` function:
```python
def check_if_model_is_supported(model: "PreTrainedModel") -> None:
    """
    Checks if the provided model is supported and raises an exception if it's not.

    Args:
        model: PreTrainedModel instance (required)

    Raises:
        NotImplementedError: If the model is not supported, with a list of supported models
    """

    # Check if model is supported and raise an error if it's not
    if not model.is_model_supported():
        supported_models = [name.encode('utf-8') for name in _SUPPORTED_MODELS]
        raise NotImplementedError(
            f"Model {model.__class__.__name__} is not supported yet, supported models: {', '.join(supported_models)}"
        )
```

**Key Points**

*   Takes a `PreTrainedModel` instance as input
*   Checks if the model is supported using `is_model_supported()` method
*   Raises an `NotImplementedError` with a list of supported models if the current model is not supported


key: chroma:document
string_value:
CODE:

def symbolic_trace(
    model: "PreTrainedModel",
    input_names: Optional[List[str]] = None,
    disable_check: bool = False,
    tracer_cls: Type[HFTracer] = HFTracer,
) -> GraphModule:
    """
    Performs symbolic tracing on the model.

    Args:
        model ([`PretrainedModel`]):
            The model to trace.
        input_names (`List[str]`, *optional*):
            The names of the inputs of the traced model. If unset, model.dummy_inputs.keys() are used instead.
        disable_check (`bool`, *optional*, defaults to `False`):
            If `True`, no check is done before trying to trace the model, this is mostly usesul for debugging purposes.
        tracer_cls (`Type[HFTracer]`, *optional*, defaults to `HFTracer`):
            The tracer class to use for instantiating the tracer. If unset, `HFTracer` is used instead.

    Returns:
        `torch.fx.GraphModule`: A GraphModule constructed by recording operations seen while tracing the model.

    Example:

        ```python
        from transformers.utils.fx import symbolic_trace

        traced_model = symbolic_trace(model, input_names=["input_ids", "attention_mask", "token_type_ids"])
        ```
    """
    if input_names is None:
        input_names = model.dummy_inputs.keys()

    input_names = list(input_names)
    concrete_args = get_concrete_args(model, input_names)

    if not disable_check:
        check_if_model_is_supported(model)

    if "past_key_values" in input_names and not getattr(model.config, "use_cache", False):
        logger.warning(
            "`past_key_values` were specified as input names, but model.config.use_cache = False, this might lead to "
            "unexpected behavior."
        )
    if "past_key_values" not in input_names and getattr(model.config, "use_cache", False):
        logger.warning(
            "`past_key_values` were not specified as input names, but model.config.use_cache = True. Setting "
            "model.config.use_cache = False."
        )
        model.config.use_cache = False

    # Tracing.
    tracer = tracer_cls()
    traced_graph = tracer.trace(model, concrete_args=concrete_args)
    traced = torch.fx.GraphModule(model, traced_graph)

    traced.config = model.config
    # The model class must be stored as an attribute to allow model deserialization, which uses trace, and thus
    # _generate_dummy_input, where the model class is needed.
    traced.class_for_deserialization = model.__class__
    traced.device = model.device



SUMMARY:
**Symbolic Trace Function Summary**

*   **Purpose:** Performs symbolic tracing on a `PreTrainedModel`, returning a `GraphModule` representing the traced model operations.

**Functionality:**

1.  Instantiates a tracer class (`HFTracer`) and creates a graph module from it.
2.  Records concrete function arguments for model input names using `get_concrete_args`.
3.  Performs checks on supported models before tracing, optionally disabling this check for debugging purposes.
4.  Ensures compatibility issue warnings when tracing models with cache settings.

**Key Implementation Details:**

*   Tracing model using `HFTracer` class
*   Using `torch.fx.GraphModule` to create the traced graph module
*   Setting up model configuration on the traced graph

**Notable Dependencies and Requirements:**

*   `PreTrainedModel`: The model to be traced.
*   `HFTracer`: Tracer class for symbolic tracing (default if not provided).
*   External dependencies:
    *   `torch.fx` for tensor flows
    *   `transformers.utils` for dummy inputs and logging.


key: chroma:document
string_value:
CODE:

class TFGPT2Tokenizer(metaclass=DummyObject):
    _backends = ["keras_nlp"]

    def __init__(self, *args, **kwargs):


SUMMARY:
**Summary:**

The `TFGPT2Tokenizer` class utilizes the `DummyObject` metaclass and inherits from the `requires_backends` function. The primary purpose of this class is to tokenize input data.

Key Implementation Details:

* Initializes with *args and **kwargs, which are passed to the `requires_backends` function.
* Requires the "keras_nlp" backend for initialization.

Notable Dependencies/Requirements:

* `DummyObject` metaclass
* `keras_nlp` backend

This class appears to be designed for tokenization in a TensorFlow-based framework. The `DummyObject` metaclass and "keras_nlp" backend are specific to the Keras Natural Language Processing (Keras NLP) library.


key: chroma:document
string_value:
CODE:

class ASTFeatureExtractor(metaclass=DummyObject):
    _backends = ["speech"]

    def __init__(self, *args, **kwargs):


SUMMARY:
**Summary:**

The `ASTFeatureExtractor` class is an implementation of the `DummyObject` metaclass, specialized for extracting features from speech data using the `_backends` backend. It initializes with required backends specified at instantiation (`requires_backends` method).

**Key Details:**

* Implements `DummyObject` metaclass.
* Specialized for extracting speech features.
* Initializes with required `["speech"]` backends.

**Dependencies/Requirements:**

* None explicitly listed, but `_backends` suggests the `requires_backends` utility is used for backend configuration.


key: chroma:document
string_value:
CODE:

class Speech2TextFeatureExtractor(metaclass=DummyObject):
    _backends = ["speech"]

    def __init__(self, *args, **kwargs):


SUMMARY:
**Summary:**

This is a Python class `Speech2TextFeatureExtractor` that inherits from the `DummyObject` metaclass. It serves as a backend for text feature extraction using speech data.

**Main Purpose and Functionality:**
The primary purpose of this class is to leverage Speech-to-Text functionality to generate features for further analysis or modeling.

**Key Implementation Details:**

* The constructor (`__init__`) requires at least one backend ("speech") to function.
* Uses the `requires_backends` function to verify the required backend presence.

**Notable Dependencies and Requirements:**

* Requires Speech-to-Text functionality with the specified backend ("speech").

No explicit external dependencies are listed in this code snippet.


key: chroma:document
string_value:
CODE:

def flex_attention_forward(
    module: torch.nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    scaling: Optional[float] = None,
    softcap: Optional[float] = None,
    **kwargs,
) -> Tuple[torch.Tensor, torch.Tensor]:
    causal_mask = attention_mask
    if causal_mask is not None:
        causal_mask = causal_mask[:, :, :, : key.shape[-2]]

    def causal_mod(score, b, h, q_idx, kv_idx):
        if softcap is not None:
            score = softcap * torch.tanh(score / softcap)
        if causal_mask is not None:
            score = score + causal_mask[b][0][q_idx][kv_idx]
        return score

    attn_output, attention_weights = flex_attention(
        query,
        key,
        value,
        score_mod=causal_mod,
        enable_gqa=True,
        scale=scaling,
        # Last time checked on PyTorch == 2.5.1: Flex Attention always computes the lse regardless.
        # For simplification, we thus always return it as no additional computations are introduced.
        return_lse=True,
    )
    # lse is returned in float32
    attention_weights = attention_weights.to(value.dtype)
    attn_output = attn_output.transpose(1, 2).contiguous()



SUMMARY:
**Functional Summary: `flex_attention_forward`**

* **Purpose:** Computes the output and weights of the Flex Attention mechanism in PyTorch.

**Main Functionality:**

* Takes in `query`, `key`, and `value` tensors, as well as optional input `attention_mask` and scaling factor.
* Applies a causal mask to `attention_mask` if provided, reducing the spatial dimensions of the tensor by `key.shape[-2]`.
* Defines a custom scoring function `causal_mod` that introduces soft attention and causal masking constraints.
* Calls the `flex_attention` function with the scoring function and other parameters.

**Key Implementation Details:**

* Involves computing the output (`attn_output`) and weights (`attention_weights`) using the `flex_attention` function.
* Returns transposed, contiguous tensor for `attn_output`.
* Returns weights in type of `value.dtype`.

**Notable Dependencies/Requirements:**

* Uses PyTorch 2.5.1 (last checked).
* Inheritance from `torch.nn.Module` via `module`.
* No explicit external dependencies listed.
* Assumes availability of `flex_attention` function.
* `scale` and `softcap` parameters accepted, defaulting to None.


key: chroma:document
string_value:
CODE:

class TorchExportableModuleWithStaticCache(torch.nn.Module):
    """
    A wrapper module designed to make a `PreTrainedModel` exportable with `torch.export`,
    specifically for use with static caching. This module ensures that the exported model
    is compatible with further lowering and execution in `ExecuTorch`.

    Note:
        This class is specifically designed to support export process using `torch.export`
        in a way that ensures the model can be further lowered and run efficiently in `ExecuTorch`.
    """

    def __init__(self, model: PreTrainedModel):
        """
        Initializes the wrapper module with the pretrained model.

        Args:
            model (`PreTrainedModel`): The pretrained model to wrap. The model must have caching
            enabled and use a 'static' caching implementation.

        Raises:
            AssertionError: If the pretrained model does not have caching enabled or if it does
            not use a 'static' caching implementation in `model.generation_config`.
        """
        super().__init__()

        # Sanity checks
        if model.generation_config is None:
            raise AssertionError(
                "The model must have a generation config to be exported with static caching. "
                "Please set `generation_config`."
            )

        if not model.generation_config.use_cache:
            raise AssertionError(
                "The model must have caching enabled to be exported with static caching. "
                "Please set `generation_config.use_cache=True`."
            )

        if model.generation_config.cache_implementation != "static":
            raise AssertionError(
                "The model must use a 'static' caching implementation to be exported with static caching. "
                "Please set `generation_config.cache_implementation='static'`."
            )

        self.model = model
        self.static_cache = StaticCache(
            config=self.model.config,
            batch_size=self.model.generation_config.cache_config.batch_size,
            max_cache_len=self.model.generation_config.cache_config.max_cache_len,
            dtype=self.model.dtype,
        )
        self.is_causal = any("CausalLM" in arch for arch in self.model.config.architectures)
        if self.is_causal:
            causal_mask = torch.tril(
                torch.ones(
                    self.static_cache.max_cache_len,
                    self.static_cache.max_cache_len,
                    dtype=torch.bool,
                )
            )
            self.register_buffer("mask", causal_mask, persistent=False)

    def forward(self, input_ids: torch.Tensor, cache_position: torch.Tensor):
        """
        Forward pass of the module, which is compatible with the ExecuTorch runtime.

        Args:
            input_ids (`torch.Tensor`): Tensor representing current input token id to the module.
            cache_position (`torch.Tensor`): Tensor representing current input position in the cache.

        Returns:
            torch.Tensor: Logits output from the model.

        This forward adapter serves two primary purposes:

        1. **Making the Model `torch.export`-Compatible**:
            The adapter hides unsupported objects, such as the `Cache`, from the graph inputs and outputs,
            enabling the model to be exportable using `torch.export` without encountering issues.

        2. **Ensuring Compatibility with `ExecuTorch` runtime**:
            The adapter matches the model's forward signature with that in `executorch/extension/llm/runner`,
            ensuring that the exported model can be executed in `ExecuTorch` out-of-the-box.
        """
        _, seqlen = input_ids.shape
        attn_mask = self.mask[cache_position, :seqlen] if self.is_causal else None
        outs = self.model(
            input_ids=input_ids,
            attention_mask=attn_mask,
            position_ids=cache_position.unsqueeze(0),
            cache_position=cache_position,
            past_key_values=self.static_cache,
            use_cache=True,
        )
        return outs.logits

    @staticmethod
    def generate(
        exported_program: torch.export.ExportedProgram, prompt_token_ids: torch.Tensor, max_new_tokens: int
    ) -> torch.Tensor:
        """
        Generate a sequence of tokens using an exported program.

        This util function is designed to test exported models by simulating the generation process.
        It processes the input prompt tokens sequentially (no parallel prefill).
        This generate function is not intended to replace the original `generate` method, and the support
        for leveraging the original `generate` is potentially planed!

        Args:
            exported_program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`.
            prompt_token_ids (`torch.Tensor`): Tensor representing the input prompt token IDs.
            max_new_tokens (`int`): Maximum number of new tokens to generate. Note that the total generation
                length is limited by both `max_new_tokens` and the model's cache size.

        Returns:
            torch.Tensor: A tensor containing the generated sequence of token IDs, including the original prompt tokens.
        """
        prompt_token_len = prompt_token_ids.shape[-1]
        max_generation_length = prompt_token_len + max_new_tokens
        for buffer_name, buffer in exported_program.named_buffers():
            if buffer_name.startswith("static_cache.key_cache"):
                max_cache_len = buffer.shape[2]
                max_generation_length = min(max_generation_length, max_cache_len)
                break

        response_tokens = []
        for input_pos in range(min(max_generation_length, prompt_token_len)):
            result = exported_program.module().forward(
                input_ids=prompt_token_ids[:, input_pos : input_pos + 1],
                cache_position=torch.tensor([input_pos], dtype=torch.long),
            )
            response_tokens.append(prompt_token_ids[0][input_pos].item())

        current_token = torch.argmax(result[:, -1, :], dim=-1).item()
        response_tokens.append(current_token)

        while len(response_tokens) < max_generation_length:
            result = exported_program.module().forward(
                input_ids=torch.tensor([[current_token]], dtype=torch.long),
                cache_position=torch.tensor([len(response_tokens)], dtype=torch.long),
            )
            current_token = torch.argmax(result[:, -1, :], dim=-1).item()
            response_tokens.append(current_token)



SUMMARY:
Here's a concise summary of the provided class code chunk:

**TorchExportableModuleWithStaticCache**

### Purpose and Functionality

This is a wrapper module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for use with static caching. It ensures that the exported model can be further lowered and executed efficiently in `ExecuTorch`.

### Key Implementation Details

* Exposes a forward method compatible with `ExecuTorch` runtime.
* Uses static caching by storing cached values in `self.static_cache`.
* Implements special masking for causal models.

### Notable Dependencies and Requirements

* `PreTrainedModel`
* `StaticCache`
* `torch.export`

Key Algorithm/Pattern:

* Static caching using `StaticCache` class.
* Masking for causal models using tril matrix.


key: chroma:document
string_value:
CODE:

def convert_and_export_with_cache(
    model: PreTrainedModel,
    example_input_ids: torch.Tensor = None,
    example_cache_position: torch.Tensor = None,
):
    """
    Convert a `PreTrainedModel` into an exportable module and export it using `torch.export`,
    ensuring the exported model is compatible with `ExecuTorch`.

    Args:
        model (`PreTrainedModel`): The pretrained model to be exported.
        example_input_ids (`torch.Tensor`): Example input token id used by `torch.export`.
        example_cache_position (`torch.Tensor`): Example current cache position used by `torch.export`.

    Returns:
        Exported program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`.
    """

    if not is_torch_greater_or_equal_than_2_3:
        raise ImportError("torch >= 2.3 is required.")

    import torch.export._trace

    with torch.no_grad():
        # TODO: The default inputs only work for text models. We need to add support for vision/audio models.
        example_input_ids = (
            example_input_ids if example_input_ids is not None else torch.tensor([[1]], dtype=torch.long)
        )
        example_cache_position = (
            example_cache_position if example_cache_position is not None else torch.tensor([0], dtype=torch.long)
        )

        # Due to issue https://github.com/pytorch/pytorch/issues/128394, we need to switch to use an internal
        # export API and pre_dispatch=False. Switch to use the public API once the issue is included in 2.5 release.
        exported_program = torch.export._trace._export(
            TorchExportableModuleWithStaticCache(model),
            args=(example_input_ids,),
            kwargs={"cache_position": example_cache_position},
            pre_dispatch=False,
            strict=True,
        )


SUMMARY:
**Function Summary**

`convert_and_export_with_cache`

Converts a `PreTrainedModel` into an exportable module using `torch.export`, ensuring compatibility with `ExecuTorch`. The function ensures required torch version (2.3+) is met.

**Key Implementation Details**

* Utilizes internal export API due to GitHub issue 128394
* Uses `pre_dispatch=False` and `strict=True`
* Specifies example input IDs and cache position using default values or provided inputs

**Notable Dependencies and Requirements**

* `PreTrainedModel` class from the trained model
* `torchexport.ExportedProgram` interface for exported program
* `TorchExportableModuleWithStaticCache` function related to PyTorch export API


key: chroma:document
string_value:
CODE:
# Integration functions:
def is_wandb_available():
    # any value of WANDB_DISABLED disables wandb
    if os.getenv("WANDB_DISABLED", "").upper() in ENV_VARS_TRUE_VALUES:
        logger.warning(
            "Using the `WANDB_DISABLED` environment variable is deprecated and will be removed in v5. Use the "
            "--report_to flag to control the integrations used for logging result (for instance --report_to none)."
        )
        return False


SUMMARY:
**Summary:**
```python
is_wandb_available() function:

Returns True if the Wandboard (WANDB) library is available for import, indicating its functionality is enabled. Disables WANDB when the "WANDB_DISABLED" environment variable is set to a truthy value.
```

**Key Implementation Details:**

* `importlib.util.find_spec("wandb")` is used to check if the Wandboard library is installed and accessible.

**Notable Dependencies or Requirements:**

* `os.getenv("WANDB_DISABLED")`: Retrieves the value of the "WANDB_DISABLED" environment variable.
* `ENV_VARS_TRUE_VALUES`: Not explicitly defined in this snippet, but likely a list of allowed values for `WANDB_DISABLED`.
* `importlib.util.find_spec`: Used to check if a library (in this case, Wandboard) is available for import.


key: chroma:document
string_value:
CODE:

def is_clearml_available():


SUMMARY:
**Summary**

Check if the `ClearML` library is installed.

**Key Implementation Details**

* Verifies the existence of an implementation-specific entry point for `ClearML`.
* Uses `importlib.util.find_spec()` to detect the presence of `clearml` as a package.

**Notable Dependencies/Requirements**

* None explicitly stated; assumes `python-importlib` and potentially other packages are available.


key: chroma:document
string_value:
CODE:

def is_comet_available():
    if os.getenv("COMET_MODE", "").upper() == "DISABLED":
        logger.warning(
            "Using the `COMET_MODE=DISABLED` environment variable is deprecated and will be removed in v5. Use the "
            "--report_to flag to control the integrations used for logging result (for instance --report_to none)."
        )
        return False

    if _is_comet_installed is False:
        return False

    if _is_comet_recent_enough is False:
        logger.warning(
            "comet_ml version %s is installed, but version %s or higher is required. "
            "Please update comet_ml to the latest version to enable Comet logging with pip install 'comet-ml>=%s'.",
            _comet_version,
            _MIN_COMET_VERSION,
            _MIN_COMET_VERSION,
        )
        return False

    if _is_comet_configured is False:
        logger.warning(
            "comet_ml is installed but the Comet API Key is not configured. "
            "Please set the `COMET_API_KEY` environment variable to enable Comet logging. "
            "Check out the documentation for other ways of configuring it: "
            "https://www.comet.com/docs/v2/guides/experiment-management/configure-sdk/#set-the-api-key"
        )
        return False



SUMMARY:
**Concise Summary**

`is_comet_available()` checks if Comet logging is enabled and configured correctly.

**Main Purpose and Functionality:**
The function verifies that the `COMET_MODE` environment variable has not been set to "DISABLED", and that `comet-ml` version is recent enough (>= `_MIN_COMET_VERSION`) with a valid API key configuration. If any condition fails, it returns `False`. Otherwise, it returns `True`.

**Key Implementation Details:**

* Verifies `COMET_MODE` environment variable
* Checks `comet-ml` version compatibility
* Requires valid API key configuration

**Notable Dependencies or Requirements:**

* `os` and `logger`
* `comet-ml` library (with specific version requirements)
* `comet_api_key` environment variable


key: chroma:document
string_value:
CODE:

def is_tensorboard_available():


SUMMARY:
**Summary**

`is_tensorboard_available()` function checks if either `tensorboard` or `tensorboardX` libraries are installed. It uses `importlib.util.find_spec` to detect the availability of these libraries.

**Key Details:**

* **Primary Purpose**: Check if required TensorBoard library exists
* **Implementation Details**:
	+ Uses `importlib.util.find_spec` to check for `tensorboard`
	+ Checks for `tensorboardX` separately using `find_spec`
* **Notable Dependencies/Requirements**:
	+ `tensorboard`
	+ `tensorboardX`
* **External Dependencies**: None


key: chroma:document
string_value:
CODE:

def is_optuna_available():


SUMMARY:
**Summary:**

`is_optuna_available()` Function:

* **Purpose:** Retrieves whether the Optuna library is available for import.

**Key Implementation Details:**

* Uses `importlib.util.find_spec` to check if Optuna spec exists.

* Returns `True` if Optuna spec found, indicating availability.

**Notable Dependencies or Requirements:**

* None. No explicit dependencies are listed.
* Implicitly relies on Python's built-in `importlib` module.
* Assumes access to the global Python namespace where `optuna` modules would be imported from.


key: chroma:document
string_value:
CODE:

def is_ray_available():


SUMMARY:
**Summary:**

 `is_ray_available()` Function:
 
 Checks if the Ray library is available for import by attempting to find its package specification using `importlib.util.find_spec("ray")`. Returns `True` if the package is found, indicating successful installation or availability; otherwise, returns `False`.

Key Implementation Detail:

* Utilizes `importlib.util.find_spec()` to probe for the presence of Ray's package specification.
 
Notable Dependency/Requirements:
 
* Ray library: The function relies on the availability of the "ray" package, implying its external dependency.
* No explicit interfaces or class relationships are applicable in this concise code snippet.


key: chroma:document
string_value:
CODE:

def is_ray_tune_available():
    if not is_ray_available():
        return False


SUMMARY:
**Summary**

`is_ray_tune_available()` checks if Ray Tune is available for installation. The function:

*   Verifies that the underlying `Ray` library is installed and available using `is_ray_available()`.
*   Tries to import the `ray.tune` module using `importlib.util.find_spec("ray.tune")`.
*   Returns `False` if Ray is not available, otherwise returns `True` indicating availability of Ray Tune.

**Key implementation details**

*   Uses `importlib.util.find_spec()` to attempt importing the `ray.tune` module.
*   Relies on the existence of a `Ray` installation for successful import.

**Notable dependencies and requirements**

*   Requires the base `Ray` library.
*   Uses `importlib`, indicating Python's built-in import mechanics are used.


key: chroma:document
string_value:
CODE:

def is_sigopt_available():


SUMMARY:
**Summary**

### Functionality
The `is_sigopt_available()` function checks if the SigOpt library is installed and available in the Python environment.

### Implementation Details
- It uses the `importlib.util.find_spec()` function to check if a spec for "sigopt" exists.

### Dependencies/Requirements
- SigOpt library: A dependency required to be installed in the Python environment.

```markdown
# Function Name: is_sigopt_available()
# Purpose: Checks if SigOpt library is available.
# Implementation Details:
  - Uses importlib.util.find_spec() function.
# Dependencies:
  - SigOpt library (installed in Python environment)
```

### Suggested Analysis Points for RAG System

- Method Signature: `is_sigopt_available()`
- Interface/Pattern Used: Dependency check using `importlib.util.find_spec()`
- Relevant Classes/Functions: `SigOpt` (library dependency)


key: chroma:document
string_value:
CODE:

def is_azureml_available():
    if importlib.util.find_spec("azureml") is None:
        return False
    if importlib.util.find_spec("azureml.core") is None:
        return False


SUMMARY:
**Summary**

`is_azureml_available()` Function Summary

* **Purpose**: Check if Azure ML is installed and accessible in the Python environment.
* **Key Implementation Details**:
	+ Checks for availability of Azure ML module (`azureml`) using `importlib.util.find_spec()`.
	+ Verifies presence of `azureml.core` sub-module within the loaded Azure ML module.
	+ Confirms existence of `run()` function within `azureml.core.run` to ensure full functionality.

**Notes**: This implementation relies on Python's built-in `importlib` module for detecting installed packages and their availability in the current environment.


key: chroma:document
string_value:
CODE:

def is_mlflow_available():
    if os.getenv("DISABLE_MLFLOW_INTEGRATION", "FALSE").upper() == "TRUE":
        return False


SUMMARY:
**Summary:**

This function checks if MLFlow integration is enabled in the environment. It returns `True` only when MLFlow is not explicitly disabled and a spec for "mlflow" can be found, indicating its availability.

**Key Implementation Details:**

- Uses `os.getenv()` to retrieve environment variable `"DISABLE_MLFLOW_INTEGRATION"`
- Returns `False` if the environment variable is set to `"TRUE"`

**Notable Dependencies and Requirements:**

- No external dependencies
- Uses `importlib.util`, which is part of Python's standard library


key: chroma:document
string_value:
CODE:

def is_dagshub_available():


SUMMARY:
**Summary**

`is_dagshub_available()` checks if the `dagshub` and `mlflow` libraries are installed by verifying their presence as executable imports. It returns `True` if either library is found, `False` otherwise.

**Key Implemented Details:**
- Checks for existence of "dagshub" using `importlib.util.find_spec`
- Checks for existence of "mlflow" using `importlib.util.find_spec`

**Notable Dependencies:**
- `importlib` module

This function relies on the `importlib` standard library, which suggests it can be used across Python environments without additional installations.


key: chroma:document
string_value:
CODE:

def is_neptune_available():


SUMMARY:
**Summary:**

`is_neptune_available` Function:

The `is_neptune_available` function checks the availability of Neptune by returning the result of `_has_neptune`. The primary purpose of this function is to provide a boolean indication of Neptune's availability, likely for configuration or control purposes.

**Implementation Details:**

* Returns the value of `_has_neptune`
* No algorithmic logic is applied; it simply returns a cached state

**Notable Dependencies/Requirements:**

* `_has_neptune` (variable)
* None


key: chroma:document
string_value:
CODE:

def is_codecarbon_available():


SUMMARY:
**Summary**

This function checks if the `codecarbon` library is installed by attempting to find its specification using `importlib.util.find_spec`. The presence of a non-None result indicates successful installation.

**Key Details:**

* Main purpose: Verify availability of `codecarbon` library.
* Implementation Details:
	+ Uses `importlib.util.find_spec("codecarbon")`
* Notable Dependencies/Requirements: 
	+ codecarbon (library)

**Output:** Boolean indicating whether the "codecarbon" library is installed.


key: chroma:document
string_value:
CODE:

def is_flytekit_available():


SUMMARY:
**Summary:**

Checks if the FlyteKit library is available for import using `importlib.util.find_spec`.

**Key Points:**

* Function Name: `is_flytekit_available`
* Functionality: Checks if FlyteKit is installed and can be imported
* Key Implementation: Uses `importlib.util.find_spec` to detect Python module existence
* Primary Purpose: To determine availability of FlyteKit for dynamic imports

**Code Highlights:**

* Method Signature: `is_flytekit_available()`
* Import Detection Condition: `importlib.util.find_spec("flytekit") is not None`

Note: This summary focuses on the specific functionality and implementation details of the provided code chunk.


key: chroma:document
string_value:
CODE:

def is_flyte_deck_standard_available():
    if not is_flytekit_available():
        return False


SUMMARY:
**Summary:**

Function `is_flyte_deck_standard_available()` checks the availability of Flyte Deck standard.
 
 **Key Insights:**
 - Returns True if Flyte Kit and specific plugin (`flytekitplugins.deck`) are available.

 **Implementation Details:**
 - Calls `is_flytekit_available()` to verify Flyte Kit availability.
 - Uses `importlib.util.find_spec()` to search for plugin specification.

**Notable Dependencies:**

*   `is_flytekit_available()` (Flyte Kit availability check)
*   `flytekitplugins.deck` (Plugin being searched)


key: chroma:document
string_value:
CODE:

def is_dvclive_available():


SUMMARY:
**Summary**

Checks if `dvclive` module is available for import.

*   **Functionality:** Verifies the presence of `dvclive` module using `importlib.util.find_spec`.
*   **Implementation Details:**
    *   No custom implementation, relying on built-in Python functionality.
    *   No specific algorithm or pattern used; it's a simple existence check.
*   **Notable Dependencies/Requirements:**  
    *   `importlib.util`: Used to find the specified module spec.
    *   `dvclive` (module): The subject of theAvailability check.

**RAG Summary Key:**

*   Purpose: Existence Check
*   Functionality: Module Availability Verification
*   Implementation Details: No Custom Algorithm/Pattern


key: chroma:document
string_value:
CODE:

def hp_params(trial):
    if is_optuna_available():
        import optuna

        if isinstance(trial, optuna.trial.BaseTrial):
            return trial.params
    if is_ray_tune_available():
        if isinstance(trial, dict):
            return trial

    if is_sigopt_available():
        if isinstance(trial, dict):
            return trial

    if is_wandb_available():
        if isinstance(trial, dict):
            return trial



SUMMARY:
**Summary:**

This Python function `hp_params` determines the type and returns parameters of a given `trial` object, which is expected to be an optimization trial from one of several supported libraries (Optuna, Ray Tune, SigOpt, or WandB). 

**Parameters and Return Value:**
- Trial object (`trial`): Input to determine its type and return parameters.

**Notable Implementation Details:**

* This function utilizes conditional checks to identify the origin of the `trial` object.
* It uses library-specific checks (`is_optuna_available`, etc.) and tries to return the corresponding trial's parameters if available.

**Dependencies/Requirements:**
None mentioned directly. However, it is assumed that a library is available on import (e.g., Optuna), but none are explicitly listed as dependencies.
- `optuna` - dependency for specific trials
- Ray Tune, SigOpt, WandB - dependency if trial object matches specific interfaces.


key: chroma:document
string_value:
CODE:

def run_hp_search_optuna(trainer, n_trials: int, direction: str, **kwargs) -> BestRun:
    import optuna
    from accelerate.utils.memory import release_memory

    if trainer.args.process_index == 0:

        def _objective(trial: optuna.Trial, checkpoint_dir=None):
            checkpoint = None
            if checkpoint_dir:
                for subdir in os.listdir(checkpoint_dir):
                    if subdir.startswith(PREFIX_CHECKPOINT_DIR):
                        checkpoint = os.path.join(checkpoint_dir, subdir)
            trainer.objective = None
            if trainer.args.world_size > 1:
                if trainer.args.parallel_mode != ParallelMode.DISTRIBUTED:
                    raise RuntimeError("only support DDP optuna HPO for ParallelMode.DISTRIBUTED currently.")
                trainer.hp_space(trial)
                fixed_trial = optuna.trial.FixedTrial(trial.params, trial.number)
                trial_main_rank_list = [fixed_trial]
                torch.distributed.broadcast_object_list(trial_main_rank_list, src=0)
                trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
            else:
                trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
            # If there hasn't been any evaluation during the training loop.
            if getattr(trainer, "objective", None) is None:
                metrics = trainer.evaluate()
                trainer.objective = trainer.compute_objective(metrics)

            # Free GPU memory
            trainer.model_wrapped, trainer.model = release_memory(trainer.model_wrapped, trainer.model)
            trainer.accelerator.clear()

            return trainer.objective

        timeout = kwargs.pop("timeout", None)
        n_jobs = kwargs.pop("n_jobs", 1)
        gc_after_trial = kwargs.pop("gc_after_trial", False)
        directions = direction if isinstance(direction, list) else None
        direction = None if directions is not None else direction
        study = optuna.create_study(direction=direction, directions=directions, **kwargs)
        study.optimize(_objective, n_trials=n_trials, timeout=timeout, n_jobs=n_jobs, gc_after_trial=gc_after_trial)
        if not study._is_multi_objective():
            best_trial = study.best_trial
            return BestRun(str(best_trial.number), best_trial.value, best_trial.params)
        else:
            best_trials = study.best_trials
            return [BestRun(str(best.number), best.values, best.params) for best in best_trials]
    else:
        for i in range(n_trials):
            trainer.objective = None
            trial_main_rank_list = [None]
            if trainer.args.parallel_mode != ParallelMode.DISTRIBUTED:
                raise RuntimeError("only support DDP optuna HPO for ParallelMode.DISTRIBUTED currently.")
            torch.distributed.broadcast_object_list(trial_main_rank_list, src=0)
            trainer.train(resume_from_checkpoint=None, trial=trial_main_rank_list[0])
            # If there hasn't been any evaluation during the training loop.
            if getattr(trainer, "objective", None) is None:
                metrics = trainer.evaluate()
                trainer.objective = trainer.compute_objective(metrics)


SUMMARY:
**Function Summary: Optuna Hyperparameter Tuning with Distributed Training**

**Primary Purpose:** Perform hyperparameter tuning using Optuna with distributed training.

* The function `run_hp_search_optuna` takes in a trainer object, number of trials (`n_trials`), direction ( optimization mode) and other keyword arguments.
* It uses the Optuna library to perform hyperparameter tuning on the given trainer object.

**Key Implementation Details:**

*   Uses `optuna.create_study()` to create a study instance with the given directions and kwargs.
*   Defines an objective function `_objective` that is used by Optuna for HPO. This function wraps around the trainer's training loop.
*   Applies distributed learning using torch.distributed broadcast_object_list for multi-GPU support.

**Notable Dependencies and Requirements:**

*   `optuna`: library for hyperparameter optimization
*   `torch`: tensor framework with distributed training capabilities (`torch.distributed`)
*   `accelerate.utils.memory`: a utility function to release GPU memory

If `trainer.args.process_index` is 0, it proceeds with the multi-GPU Optuna hp search.

Otherwise (on other GPUS), perform  n_trial iterations of single-node training without Optuna hp tunning.

There are two return types.

Best-run returns if using Optunal HPO.

None returns when running on a process index that is not zero.


key: chroma:document
string_value:
CODE:

def run_hp_search_ray(trainer, n_trials: int, direction: str, **kwargs) -> BestRun:
    import ray
    import ray.train

    def _objective(trial: dict, local_trainer):
        try:
            from transformers.utils.notebook import NotebookProgressCallback

            if local_trainer.pop_callback(NotebookProgressCallback):
                local_trainer.add_callback(ProgressCallback)
        except ModuleNotFoundError:
            pass

        local_trainer.objective = None

        checkpoint = ray.train.get_checkpoint()
        if checkpoint:
            # Upon trial resume, the local_trainer's objective gets reset to None.
            # If `local_trainer.train` is a noop (training has already reached
            # the target number of epochs/steps), then this would
            # trigger an unnecessary extra checkpoint at the end of training.
            # -> Set the objective to a dummy value upon resume as a workaround.
            local_trainer.objective = "objective"

            with checkpoint.as_directory() as checkpoint_dir:
                checkpoint_path = next(Path(checkpoint_dir).glob(f"{PREFIX_CHECKPOINT_DIR}*")).as_posix()
                local_trainer.train(resume_from_checkpoint=checkpoint_path, trial=trial)
        else:
            local_trainer.train(trial=trial)

        # If there hasn't been any evaluation during the training loop.
        if getattr(local_trainer, "objective", None) is None:
            metrics = local_trainer.evaluate()
            local_trainer.objective = local_trainer.compute_objective(metrics)

            metrics.update({"objective": local_trainer.objective, "done": True})

            with tempfile.TemporaryDirectory() as temp_checkpoint_dir:
                local_trainer._tune_save_checkpoint(checkpoint_dir=temp_checkpoint_dir)
                checkpoint = ray.train.Checkpoint.from_directory(temp_checkpoint_dir)
                ray.train.report(metrics, checkpoint=checkpoint)

    if not trainer._memory_tracker.skip_memory_metrics:
        from ..trainer_utils import TrainerMemoryTracker

        logger.warning(
            "Memory tracking for your Trainer is currently "
            "enabled. Automatically disabling the memory tracker "
            "since the memory tracker is not serializable."
        )
        trainer._memory_tracker = TrainerMemoryTracker(skip_memory_metrics=True)

    # The model and TensorBoard writer do not pickle so we have to remove them (if they exists)
    # while doing the ray hp search.
    _tb_writer = trainer.pop_callback(TensorBoardCallback)
    trainer.model = None

    # Setup default `resources_per_trial`.
    if "resources_per_trial" not in kwargs:
        # Default to 1 CPU and 1 GPU (if applicable) per trial.
        kwargs["resources_per_trial"] = {"cpu": 1}
        if trainer.args.n_gpu > 0:
            kwargs["resources_per_trial"]["gpu"] = 1
        resource_msg = "1 CPU" + (" and 1 GPU" if trainer.args.n_gpu > 0 else "")
        logger.info(
            "No `resources_per_trial` arg was passed into "
            "`hyperparameter_search`. Setting it to a default value "
            f"of {resource_msg} for each trial."
        )
    # Make sure each trainer only uses GPUs that were allocated per trial.
    gpus_per_trial = kwargs["resources_per_trial"].get("gpu", 0)
    trainer.args._n_gpu = gpus_per_trial

    # Setup default `progress_reporter`.
    if "progress_reporter" not in kwargs:
        from ray.tune import CLIReporter

        kwargs["progress_reporter"] = CLIReporter(metric_columns=["objective"])

    if "scheduler" in kwargs:
        from ray.tune.schedulers import ASHAScheduler, HyperBandForBOHB, MedianStoppingRule, PopulationBasedTraining

        # Check for `do_eval` and `eval_during_training` for schedulers that require intermediate reporting.
        if isinstance(
            kwargs["scheduler"], (ASHAScheduler, MedianStoppingRule, HyperBandForBOHB, PopulationBasedTraining)
        ) and (not trainer.args.do_eval or trainer.args.eval_strategy == IntervalStrategy.NO):
            raise RuntimeError(
                "You are using {cls} as a scheduler but you haven't enabled evaluation during training. "
                "This means your trials will not report intermediate results to Ray Tune, and "
                "can thus not be stopped early or used to exploit other trials parameters. "
                "If this is what you want, do not use {cls}. If you would like to use {cls}, "
                "make sure you pass `do_eval=True` and `eval_strategy='steps'` in the "
                "Trainer `args`.".format(cls=type(kwargs["scheduler"]).__name__)
            )

    trainable = ray.tune.with_parameters(_objective, local_trainer=trainer)

    @functools.wraps(trainable)
    def dynamic_modules_import_trainable(*args, **kwargs):
        """
        Wrapper around `tune.with_parameters` to ensure datasets_modules are loaded on each Actor.

        Without this, an ImportError will be thrown. See https://github.com/huggingface/transformers/issues/11565.

        Assumes that `_objective`, defined above, is a function.
        """
        if is_datasets_available():
            import datasets.load

            dynamic_modules_path = os.path.join(datasets.load.init_dynamic_modules(), "__init__.py")
            # load dynamic_modules from path
            spec = importlib.util.spec_from_file_location("datasets_modules", dynamic_modules_path)
            datasets_modules = importlib.util.module_from_spec(spec)
            sys.modules[spec.name] = datasets_modules
            spec.loader.exec_module(datasets_modules)
        return trainable(*args, **kwargs)

    # special attr set by tune.with_parameters
    if hasattr(trainable, "__mixins__"):
        dynamic_modules_import_trainable.__mixins__ = trainable.__mixins__

    analysis = ray.tune.run(
        dynamic_modules_import_trainable,
        config=trainer.hp_space(None),
        num_samples=n_trials,
        **kwargs,
    )
    best_trial = analysis.get_best_trial(metric="objective", mode=direction[:3], scope=trainer.args.ray_scope)
    best_run = BestRun(best_trial.trial_id, best_trial.last_result["objective"], best_trial.config, analysis)
    if _tb_writer is not None:
        trainer.add_callback(_tb_writer)


SUMMARY:
**HP Search Ray Trainer Summary**

### Main Purpose and Functionality
The `run_hp_search_ray` function implements Hyperparameter Tuning using Ray Tune, a framework for large-scale distributed optimization. It takes in a trainer object, number of trials (`n_trials`), direction (mode) of the objective function, and additional keyword arguments.

**Key Implementation Details**

*   Defines an `_objective` function that is used as a parameter to be tuned.
*   Handles checkpointing and resume functionality when training from existing checkpoints.
*   Sets up default values for `resources_per_trial`, `progress_reporter`, and other configuration options based on the provided kwargs.
*   Checks if specific schedulers require evaluation during training.
*   Uses the `tune.with_parameters` decorator to create a trainable object with the `_objective` function.

**Notable Dependencies or Requirements**

*   Import of external dependencies, such as:
    *   `ray`
    *   `ray.train`
    *   `ray.tune`
    *   `transformers.utils.notebook`
    *   `datasets.load`

The function returns a `BestRun` object containing the best trial results.


key: chroma:document
string_value:
CODE:

def run_hp_search_sigopt(trainer, n_trials: int, direction: str, **kwargs) -> BestRun:
    import sigopt

    if trainer.args.process_index == 0:
        if importlib.metadata.version("sigopt") >= "8.0.0":
            sigopt.set_project("huggingface")

            experiment = sigopt.create_experiment(
                name="huggingface-tune",
                type="offline",
                parameters=trainer.hp_space(None),
                metrics=[{"name": "objective", "objective": direction, "strategy": "optimize"}],
                parallel_bandwidth=1,
                budget=n_trials,
            )

            logger.info(f"created experiment: https://app.sigopt.com/experiment/{experiment.id}")

            for run in experiment.loop():
                with run:
                    trainer.objective = None
                    if trainer.args.world_size > 1:
                        if trainer.args.parallel_mode != ParallelMode.DISTRIBUTED:
                            raise RuntimeError("only support DDP Sigopt HPO for ParallelMode.DISTRIBUTED currently.")
                        trainer._hp_search_setup(run.run)
                        torch.distributed.broadcast_object_list(pickle.dumps(trainer.args), src=0)
                        trainer.train(resume_from_checkpoint=None)
                    else:
                        trainer.train(resume_from_checkpoint=None, trial=run.run)
                    # If there hasn't been any evaluation during the training loop.
                    if getattr(trainer, "objective", None) is None:
                        metrics = trainer.evaluate()
                        trainer.objective = trainer.compute_objective(metrics)
                    run.log_metric("objective", trainer.objective)

            best = list(experiment.get_best_runs())[0]
            best_run = BestRun(best.id, best.values["objective"].value, best.assignments)
        else:
            from sigopt import Connection

            conn = Connection()
            proxies = kwargs.pop("proxies", None)
            if proxies is not None:
                conn.set_proxies(proxies)

            experiment = conn.experiments().create(
                name="huggingface-tune",
                parameters=trainer.hp_space(None),
                metrics=[{"name": "objective", "objective": direction, "strategy": "optimize"}],
                parallel_bandwidth=1,
                observation_budget=n_trials,
                project="huggingface",
            )
            logger.info(f"created experiment: https://app.sigopt.com/experiment/{experiment.id}")

            while experiment.progress.observation_count < experiment.observation_budget:
                suggestion = conn.experiments(experiment.id).suggestions().create()
                trainer.objective = None
                if trainer.args.world_size > 1:
                    if trainer.args.parallel_mode != ParallelMode.DISTRIBUTED:
                        raise RuntimeError("only support DDP Sigopt HPO for ParallelMode.DISTRIBUTED currently.")
                    trainer._hp_search_setup(suggestion)
                    torch.distributed.broadcast_object_list(pickle.dumps(trainer.args), src=0)
                    trainer.train(resume_from_checkpoint=None)
                else:
                    trainer.train(resume_from_checkpoint=None, trial=suggestion)
                # If there hasn't been any evaluation during the training loop.
                if getattr(trainer, "objective", None) is None:
                    metrics = trainer.evaluate()
                    trainer.objective = trainer.compute_objective(metrics)

                values = [{"name": "objective", "value": trainer.objective}]
                obs = conn.experiments(experiment.id).observations().create(suggestion=suggestion.id, values=values)
                logger.info(f"[suggestion_id, observation_id]: [{suggestion.id}, {obs.id}]")
                experiment = conn.experiments(experiment.id).fetch()

            best = list(conn.experiments(experiment.id).best_assignments().fetch().iterate_pages())[0]
            best_run = BestRun(best.id, best.value, best.assignments)
        return best_run
    else:
        for i in range(n_trials):
            trainer.objective = None
            args_main_rank = list(pickle.dumps(trainer.args))
            if trainer.args.parallel_mode != ParallelMode.DISTRIBUTED:
                raise RuntimeError("only support DDP Sigopt HPO for ParallelMode.DISTRIBUTED currently.")
            torch.distributed.broadcast_object_list(args_main_rank, src=0)
            args = pickle.loads(bytes(args_main_rank))
            for key, value in asdict(args).items():
                if key != "local_rank":
                    setattr(trainer.args, key, value)
            trainer.train(resume_from_checkpoint=None)
            # If there hasn't been any evaluation during the training loop.
            if getattr(trainer, "objective", None) is None:
                metrics = trainer.evaluate()
                trainer.objective = trainer.compute_objective(metrics)


SUMMARY:
**Function Summary: sigopt_hp_search**

The `run_hp_search_sigopt` function performs hyperparameter optimization using the SigOpt library. It takes in the following parameters:

*   `trainer`: The training object
*   `n_trials`: The number of trials to run for hyperparameter searching
*   `direction`: The direction for optimization (minimization or maximization)
*   `**kwargs`: Additional keyword arguments used for connection to SigOpt

The function handles two cases based on whether the Trainer's process index is 0:
    *   **Case 1 (Trainer's process index is not 0)**:

            -   It creates an experiment using SigOpt with `name`, `parameters`, `metrics`, and other configurations set up according to parameters passed by arguments.
            -   Runs multiple iterations of the experiment (`n_trials`) to gather observational data for hyperparameter learning. This includes calling the model without any specific input, training a model, computing loss metrics and objective functions, observing these results with SigOpt's connection tools, returning optimal values.
        *   The `experiment` is used by `conn.experiments().create()` and other operations are defined to interact further using signals from SigOpt.

            -   It creates an experiment with configured parameters (`name`, `parameters`, `metrics`) set up according to the Trainer object. When a particular run is identified as best (using a list operation), it pulls out information like a specific id associated with that optimal solution, along with its value and other assignments.
*   **Case 2 (Trainer's process index not equal to 0)**:

            -   SigOpt connection was set up only and runs trials within an environment in which there are no multiple process indices.

The complete return statement results in either a `BestRun` object, containing one of many values pulled from the SigOpt search, including optimal numerical solution (which is `trainer.objective`) and assignments corresponding to each value obtained during execution.


key: chroma:document
string_value:
CODE:

def run_hp_search_wandb(trainer, n_trials: int, direction: str, **kwargs) -> BestRun:
    from ..integrations import is_wandb_available

    if not is_wandb_available():
        raise ImportError("This function needs wandb installed: `pip install wandb`")
    import wandb

    # add WandbCallback if not already added in trainer callbacks
    reporting_to_wandb = False
    for callback in trainer.callback_handler.callbacks:
        if isinstance(callback, WandbCallback):
            reporting_to_wandb = True
            break
    if not reporting_to_wandb:
        trainer.add_callback(WandbCallback())
    trainer.args.report_to = ["wandb"]
    best_trial = {"run_id": None, "objective": None, "hyperparameters": None}
    sweep_id = kwargs.pop("sweep_id", None)
    project = kwargs.pop("project", None)
    name = kwargs.pop("name", None)
    entity = kwargs.pop("entity", None)
    metric = kwargs.pop("metric", "eval/loss")

    sweep_config = trainer.hp_space(None)
    sweep_config["metric"]["goal"] = direction
    sweep_config["metric"]["name"] = metric
    if name:
        sweep_config["name"] = name

    def _objective():
        run = wandb.run if wandb.run else wandb.init()
        trainer.state.trial_name = run.name
        run.config.update({"assignments": {}, "metric": metric})
        config = wandb.config

        trainer.objective = None

        trainer.train(resume_from_checkpoint=None, trial=vars(config)["_items"])
        # If there hasn't been any evaluation during the training loop.
        if getattr(trainer, "objective", None) is None:
            metrics = trainer.evaluate()
            trainer.objective = trainer.compute_objective(metrics)
            format_metrics = rewrite_logs(metrics)
            if metric not in format_metrics:
                logger.warning(
                    f"Provided metric {metric} not found. This might result in unexpected sweeps charts. The available"
                    f" metrics are {format_metrics.keys()}"
                )
        best_score = False
        if best_trial["run_id"] is not None:
            if direction == "minimize":
                best_score = trainer.objective < best_trial["objective"]
            elif direction == "maximize":
                best_score = trainer.objective > best_trial["objective"]

        if best_score or best_trial["run_id"] is None:
            best_trial["run_id"] = run.id
            best_trial["objective"] = trainer.objective
            best_trial["hyperparameters"] = dict(config)

        return trainer.objective

    sweep_id = wandb.sweep(sweep_config, project=project, entity=entity) if not sweep_id else sweep_id
    logger.info(f"wandb sweep id - {sweep_id}")
    wandb.agent(sweep_id, function=_objective, count=n_trials)



SUMMARY:
**Summary:**

`hp_search` Function
=====================

**Purpose and Functionality:**
This function runs a Hyperparameter Search (HPSearch) using the WandB library, aiming to optimize the objective metric in a given model. 

### Main Process

The main process performs the following steps:

*   Initializes WandB if not available.
*   Registers a WandB callback with the trainer.
*   Prepares HP space configuration for modeling hyperparameters.
*   Runs a new trial in the specified sweep and evaluates it. 
*   Stores results, including objective metric and hyperparameters, into best_run.

### Implementation Details

#### Key Algorithms/Patrones

-   WandB S sweeps for model training with hyperparameter tuning
-   Objective function and evaluation of models using `compute_objective` 

#### Notable Dependencies and Requirements

-   WandB installation (verified)

### Method Signatures

-   `_objective`: WandB task for trial execution 
    `- Wandb.run` or `wandb.init` depends on the availability.
-   `evaluate`, `train`, `compute_objective`, `rewrite_logs`

#### External Dependencies

-   WandB library
-   Trainer and model dependencies through `trainer.args`


key: chroma:document
string_value:
CODE:

def get_available_reporting_integrations():
    integrations = []
    if is_azureml_available() and not is_mlflow_available():
        integrations.append("azure_ml")
    if is_comet_available():
        integrations.append("comet_ml")
    if is_dagshub_available():
        integrations.append("dagshub")
    if is_dvclive_available():
        integrations.append("dvclive")
    if is_mlflow_available():
        integrations.append("mlflow")
    if is_neptune_available():
        integrations.append("neptune")
    if is_tensorboard_available():
        integrations.append("tensorboard")
    if is_wandb_available():
        integrations.append("wandb")
    if is_codecarbon_available():
        integrations.append("codecarbon")
    if is_clearml_available():
        integrations.append("clearml")


SUMMARY:
**Summary:**

This function, `get_available_reporting_integrations`, returns a list of available reporting integration names. The function checks for the availability of various Machine Learning (ML) frameworks and libraries using helper functions (`is_azureml_available()`, etc.), and appends corresponding integration names to the result list.

**Key Implementation Details:**

* Iterates through multiple ML frameworks and libraries to determine availability.
* Helper functions used:
	+ `is_<framework_name>_available()` for individual framework checks (e.g., `is_azureml_available()`).
* Appends integration names to the result list based on availability checks.
* No complex algorithms or patterns applied.

**Notable Dependencies:**

None noted. 

However, implicit dependencies may exist:
- The `is_<framework_name>_available()` helper functions likely rely on external libraries or frameworks (e.g., Azure ML API).
- This function does not import any external libraries; however, the underlying framework-specific checks might.


key: chroma:document
string_value:
CODE:

def rewrite_logs(d):
    new_d = {}
    eval_prefix = "eval_"
    eval_prefix_len = len(eval_prefix)
    test_prefix = "test_"
    test_prefix_len = len(test_prefix)
    for k, v in d.items():
        if k.startswith(eval_prefix):
            new_d["eval/" + k[eval_prefix_len:]] = v
        elif k.startswith(test_prefix):
            new_d["test/" + k[test_prefix_len:]] = v
        else:
            new_d["train/" + k] = v


SUMMARY:
Here is a concise and focused summary of the function code chunk:

**Function Summary**

* `rewrite_logs(d)`: Reorganizes input dictionary `d` into a structured format.

**Primary Purpose**
Reorganize logs from evaluation (`eval_`), testing (`test_`), and training datasets into distinct categories (`train`, `eval/test`, and `test`) in the output dictionary.

**Key Implementation Details**

* Uses prefix matching to categorize input keys.
* Utilizes string slicing to extract relevant prefixes from input keys.

**Notable Dependencies or Requirements**
None listed.

This summary aims to distill the function's main purpose, key implementation details, and any notable dependencies into a compact and understandable format.


key: chroma:document
string_value:
CODE:

class TensorBoardCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that sends the logs to [TensorBoard](https://www.tensorflow.org/tensorboard).

    Args:
        tb_writer (`SummaryWriter`, *optional*):
            The writer to use. Will instantiate one if not set.
    """

    def __init__(self, tb_writer=None):
        has_tensorboard = is_tensorboard_available()
        if not has_tensorboard:
            raise RuntimeError(
                "TensorBoardCallback requires tensorboard to be installed. Either update your PyTorch version or"
                " install tensorboardX."
            )
        if has_tensorboard:
            try:
                from torch.utils.tensorboard import SummaryWriter  # noqa: F401

                self._SummaryWriter = SummaryWriter
            except ImportError:
                try:
                    from tensorboardX import SummaryWriter

                    self._SummaryWriter = SummaryWriter
                except ImportError:
                    self._SummaryWriter = None
        else:
            self._SummaryWriter = None
        self.tb_writer = tb_writer

    def _init_summary_writer(self, args, log_dir=None):
        log_dir = log_dir or args.logging_dir
        if self._SummaryWriter is not None:
            self.tb_writer = self._SummaryWriter(log_dir=log_dir)

    def on_train_begin(self, args, state, control, **kwargs):
        if not state.is_world_process_zero:
            return

        log_dir = None

        if state.is_hyper_param_search:
            trial_name = state.trial_name
            if trial_name is not None:
                log_dir = os.path.join(args.logging_dir, trial_name)

        if self.tb_writer is None:
            self._init_summary_writer(args, log_dir)

        if self.tb_writer is not None:
            self.tb_writer.add_text("args", args.to_json_string())
            if "model" in kwargs:
                model = kwargs["model"]
                if hasattr(model, "config") and model.config is not None:
                    model_config_json = model.config.to_json_string()
                    self.tb_writer.add_text("model_config", model_config_json)

    def on_log(self, args, state, control, logs=None, **kwargs):
        if not state.is_world_process_zero:
            return

        if self.tb_writer is None:
            self._init_summary_writer(args)

        if self.tb_writer is not None:
            logs = rewrite_logs(logs)
            for k, v in logs.items():
                if isinstance(v, (int, float)):
                    self.tb_writer.add_scalar(k, v, state.global_step)
                elif isinstance(v, str):
                    self.tb_writer.add_text(k, v, state.global_step)
                else:
                    logger.warning(
                        "Trainer is attempting to log a value of "
                        f'"{v}" of type {type(v)} for key "{k}" as a scalar. '
                        "This invocation of Tensorboard's writer.add_scalar() "
                        "is incorrect so we dropped this attribute."
                    )
            self.tb_writer.flush()

    def on_train_end(self, args, state, control, **kwargs):
        if self.tb_writer:
            self.tb_writer.close()


SUMMARY:
**TensorBoardCallback Class Summary**
=====================================

### Purpose and Functionality

The `TensorBoardCallback` class is a custom callback that sends Tensorflow logging data to [TensorBoard](https://www.tensorflow.org/tensorboard). It allows users to visualize training data in a human-readable format.

### Key Implementation Details

* Initializes the callback with an optional `SummaryWriter` instance or creates one if not provided.
* Checks for availability of `tensorboardX` and raises an error if it is not installed.
* Overwrites the logging directory based on user input.

### Notable Dependencies and Requirements

* `torch.utils.tensorboard`: Used to create a `SummaryWriter` instance.
* `tensorboardX`: Used as a fallback in case `torch.utils.tensorboard` is not available.
* `TensorBoard`: Required for visualization of training data.


key: chroma:document
string_value:
CODE:

def save_model_architecture_to_file(model: Any, output_dir: str):
    with open(f"{output_dir}/model_architecture.txt", "w+") as f:
        if isinstance(model, PreTrainedModel):
            print(model, file=f)
        elif is_tf_available() and isinstance(model, TFPreTrainedModel):

            def print_to_file(s):
                print(s, file=f)

            model.summary(print_fn=print_to_file)
        elif is_torch_available() and (
            isinstance(model, (torch.nn.Module, PushToHubMixin)) and hasattr(model, "base_model")
        ):


SUMMARY:
**Function Summary: save_model_architecture_to_file**

**Purpose:** Saves the architecture of a given machine learning model to a text file.

**Primary Functionality:** Writes the model's configuration to a file in the specified output directory.

**Implementation Details:**

* Supports three types of models:
	+ PreTrainedModel
	+ TFPreTrainedModel (uses TensorFlow as backend)
	+ PushToHubMixin (uses PyTorch and Hugging Face Hub) 
* For each model type, prints its configuration to the file using a custom printing function.
* Utilizes platform-dependent functionality (e.g., TensorFlow's `summary()` method).

**Dependencies and Requirements:**

* None specified
* Assumes availability of platforms dependent on the model backend:
	+ TensorFlow (for PreTrainedModel and TFPreTrainedModel)
	+ PyTorch (for PushToHubMixin)


key: chroma:document
string_value:
CODE:

class WandbLogModel(str, Enum):
    """Enum of possible log model values in W&B."""

    CHECKPOINT = "checkpoint"
    END = "end"
    FALSE = "false"

    @property
    def is_enabled(self) -> bool:
        """Check if the value corresponds to a state where the `WANDB_LOG_MODEL` setting is enabled."""
        return self in (WandbLogModel.CHECKPOINT, WandbLogModel.END)

    @classmethod
    def _missing_(cls, value: Any) -> "WandbLogModel":
        if not isinstance(value, str):
            raise ValueError(f"Expecting to have a string `WANDB_LOG_MODEL` setting, but got {type(value)}")
        if value.upper() in ENV_VARS_TRUE_VALUES:
            raise DeprecationWarning(
                f"Setting `WANDB_LOG_MODEL` as {os.getenv('WANDB_LOG_MODEL')} is deprecated and will be removed in "
                "version 5 of transformers. Use one of `'end'` or `'checkpoint'` instead."
            )
            logger.info(f"Setting `WANDB_LOG_MODEL` from {os.getenv('WANDB_LOG_MODEL')} to `end` instead")
            return WandbLogModel.END
        logger.warning(
            f"Received unrecognized `WANDB_LOG_MODEL` setting value={value}; so disabling `WANDB_LOG_MODEL`"
        )


SUMMARY:
**Summary:**

The `WandbLogModel` class defines an Enum with three values (`CHECKPOINT`, `END`, `FALSE`) representing possible log model states in Weights & Biases (W&B). The class provides a `is_enabled` property to check if the current value corresponds to a enabled state.

**Key Implementation Details:**

* `_missing_` method handles invalid input by:
	+ Raising a ValueError for non-string inputs
	+ Returning a deprecated warning for recognized string inputs not equal to '_checkpoint' or 'END', and logging a warning.
	+ Logging a debug message when 'TRUE' is passed, replacing it with 'END'

**Notable Dependencies:**

* `ENV_VARS_TRUE_VALUES`
* `os` module (for environment variables)


key: chroma:document
string_value:
CODE:

class WandbCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that logs metrics, media, model checkpoints to [Weight and Biases](https://www.wandb.com/).
    """

    def __init__(self):
        has_wandb = is_wandb_available()
        if not has_wandb:
            raise RuntimeError("WandbCallback requires wandb to be installed. Run `pip install wandb`.")
        if has_wandb:
            import wandb

            self._wandb = wandb
        self._initialized = False
        self._log_model = WandbLogModel(os.getenv("WANDB_LOG_MODEL", "false"))

    def setup(self, args, state, model, **kwargs):
        """
        Setup the optional Weights & Biases (*wandb*) integration.

        One can subclass and override this method to customize the setup if needed. Find more information
        [here](https://docs.wandb.ai/guides/integrations/huggingface). You can also override the following environment
        variables:

        Environment:
        - **WANDB_LOG_MODEL** (`str`, *optional*, defaults to `"false"`):
            Whether to log model and checkpoints during training. Can be `"end"`, `"checkpoint"` or `"false"`. If set
            to `"end"`, the model will be uploaded at the end of training. If set to `"checkpoint"`, the checkpoint
            will be uploaded every `args.save_steps` . If set to `"false"`, the model will not be uploaded. Use along
            with [`~transformers.TrainingArguments.load_best_model_at_end`] to upload best model.

            <Deprecated version="5.0">

            Setting `WANDB_LOG_MODEL` as `bool` will be deprecated in version 5 of 🤗 Transformers.

            </Deprecated>
        - **WANDB_WATCH** (`str`, *optional* defaults to `"false"`):
            Can be `"gradients"`, `"all"`, `"parameters"`, or `"false"`. Set to `"all"` to log gradients and
            parameters.
        - **WANDB_PROJECT** (`str`, *optional*, defaults to `"huggingface"`):
            Set this to a custom string to store results in a different project.
        - **WANDB_DISABLED** (`bool`, *optional*, defaults to `False`):
            Whether to disable wandb entirely. Set `WANDB_DISABLED=true` to disable.
        """
        if self._wandb is None:
            return
        self._initialized = True

        # prepare to handle potential configuration issues during setup
        from wandb.sdk.lib.config_util import ConfigError as WandbConfigError

        if state.is_world_process_zero:
            logger.info(
                'Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"'
            )
            combined_dict = {**args.to_dict()}

            if hasattr(model, "config") and model.config is not None:
                model_config = model.config if isinstance(model.config, dict) else model.config.to_dict()
                combined_dict = {**model_config, **combined_dict}
            if hasattr(model, "peft_config") and model.peft_config is not None:
                peft_config = model.peft_config
                combined_dict = {**{"peft_config": peft_config}, **combined_dict}
            trial_name = state.trial_name
            init_args = {}
            if trial_name is not None:
                init_args["name"] = trial_name
                init_args["group"] = args.run_name
            elif args.run_name is not None:
                init_args["name"] = args.run_name
                if args.run_name == args.output_dir:
                    self._wandb.termwarn(
                        "The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was "
                        "not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.",
                        repeat=False,
                    )

            if self._wandb.run is None:
                self._wandb.init(
                    project=os.getenv("WANDB_PROJECT", "huggingface"),
                    **init_args,
                )
            # add config parameters (run may have been created manually)
            self._wandb.config.update(combined_dict, allow_val_change=True)

            # define default x-axis (for latest wandb versions)
            if getattr(self._wandb, "define_metric", None):
                self._wandb.define_metric("train/global_step")
                self._wandb.define_metric("*", step_metric="train/global_step", step_sync=True)

            # keep track of model topology and gradients, unsupported on TPU
            _watch_model = os.getenv("WANDB_WATCH", "false")
            if not is_torch_xla_available() and _watch_model in ("all", "parameters", "gradients"):
                self._wandb.watch(model, log=_watch_model, log_freq=max(100, state.logging_steps))
            self._wandb.run._label(code="transformers_trainer")

            # add number of model parameters to wandb config
            try:
                self._wandb.config["model/num_parameters"] = model.num_parameters()
            except AttributeError:
                logger.info(
                    "Could not log the number of model parameters in Weights & Biases due to an AttributeError."
                )
            except WandbConfigError:
                logger.warning(
                    "A ConfigError was raised whilst setting the number of model parameters in Weights & Biases config."
                )

            # log the initial model architecture to an artifact
            if self._log_model.is_enabled:
                with tempfile.TemporaryDirectory() as temp_dir:
                    model_name = (
                        f"model-{self._wandb.run.id}"
                        if (args.run_name is None or args.run_name == args.output_dir)
                        else f"model-{self._wandb.run.name}"
                    )
                    model_artifact = self._wandb.Artifact(
                        name=model_name,
                        type="model",
                        metadata={
                            "model_config": model.config.to_dict() if hasattr(model, "config") else None,
                            "num_parameters": self._wandb.config.get("model/num_parameters"),
                            "initial_model": True,
                        },
                    )
                    # add the architecture to a separate text file
                    save_model_architecture_to_file(model, temp_dir)

                    for f in Path(temp_dir).glob("*"):
                        if f.is_file():
                            with model_artifact.new_file(f.name, mode="wb") as fa:
                                fa.write(f.read_bytes())
                    self._wandb.run.log_artifact(model_artifact, aliases=["base_model"])

                    badge_markdown = (
                        f'[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge'
                        f'-28.svg" alt="Visualize in Weights & Biases" width="20'
                        f'0" height="32"/>]({self._wandb.run.get_url()})'
                    )

                    modelcard.AUTOGENERATED_TRAINER_COMMENT += f"\n{badge_markdown}"

    def on_train_begin(self, args, state, control, model=None, **kwargs):
        if self._wandb is None:
            return
        hp_search = state.is_hyper_param_search
        if hp_search:
            self._wandb.finish()
            self._initialized = False
            args.run_name = None
        if not self._initialized:
            self.setup(args, state, model, **kwargs)

    def on_train_end(self, args, state, control, model=None, tokenizer=None, **kwargs):
        if self._wandb is None:
            return
        if self._log_model.is_enabled and self._initialized and state.is_world_process_zero:
            from ..trainer import Trainer

            fake_trainer = Trainer(args=args, model=model, processing_class=tokenizer, eval_dataset=["fake"])
            with tempfile.TemporaryDirectory() as temp_dir:
                fake_trainer.save_model(temp_dir)
                metadata = (
                    {
                        k: v
                        for k, v in dict(self._wandb.summary).items()
                        if isinstance(v, numbers.Number) and not k.startswith("_")
                    }
                    if not args.load_best_model_at_end
                    else {
                        f"eval/{args.metric_for_best_model}": state.best_metric,
                        "train/total_floss": state.total_flos,
                        "model/num_parameters": self._wandb.config.get("model/num_parameters"),
                    }
                )
                metadata["final_model"] = True
                logger.info("Logging model artifacts. ...")
                model_name = (
                    f"model-{self._wandb.run.id}"
                    if (args.run_name is None or args.run_name == args.output_dir)
                    else f"model-{self._wandb.run.name}"
                )
                # add the model architecture to a separate text file
                save_model_architecture_to_file(model, temp_dir)

                artifact = self._wandb.Artifact(name=model_name, type="model", metadata=metadata)
                for f in Path(temp_dir).glob("*"):
                    if f.is_file():
                        with artifact.new_file(f.name, mode="wb") as fa:
                            fa.write(f.read_bytes())
                self._wandb.run.log_artifact(artifact, aliases=["final_model"])

    def on_log(self, args, state, control, model=None, logs=None, **kwargs):
        single_value_scalars = [
            "train_runtime",
            "train_samples_per_second",
            "train_steps_per_second",
            "train_loss",
            "total_flos",
        ]

        if self._wandb is None:
            return
        if not self._initialized:
            self.setup(args, state, model)
        if state.is_world_process_zero:
            for k, v in logs.items():
                if k in single_value_scalars:
                    self._wandb.run.summary[k] = v
            non_scalar_logs = {k: v for k, v in logs.items() if k not in single_value_scalars}
            non_scalar_logs = rewrite_logs(non_scalar_logs)
            self._wandb.log({**non_scalar_logs, "train/global_step": state.global_step})

    def on_save(self, args, state, control, **kwargs):
        if self._log_model == WandbLogModel.CHECKPOINT and self._initialized and state.is_world_process_zero:
            checkpoint_metadata = {
                k: v
                for k, v in dict(self._wandb.summary).items()
                if isinstance(v, numbers.Number) and not k.startswith("_")
            }
            checkpoint_metadata["model/num_parameters"] = self._wandb.config.get("model/num_parameters")

            ckpt_dir = f"checkpoint-{state.global_step}"
            artifact_path = os.path.join(args.output_dir, ckpt_dir)
            logger.info(f"Logging checkpoint artifacts in {ckpt_dir}. ...")
            checkpoint_name = (
                f"model-{self._wandb.run.id}"
                if (args.run_name is None or args.run_name == args.output_dir)
                else f"model-{self._wandb.run.name}"
            )
            artifact = self._wandb.Artifact(name=checkpoint_name, type="model", metadata=checkpoint_metadata)
            artifact.add_dir(artifact_path)
            self._wandb.log_artifact(
                artifact, aliases=[f"epoch_{round(state.epoch, 2)}", f"checkpoint_global_step_{state.global_step}"]
            )

    def on_predict(self, args, state, control, metrics, **kwargs):
        if self._wandb is None:
            return
        if not self._initialized:
            self.setup(args, state, **kwargs)
        if state.is_world_process_zero:
            metrics = rewrite_logs(metrics)


SUMMARY:
The code snippet appears to be a part of a class that implements the interface for logging data using `Wandb`, a popular open-source library for tracking machine learning experiments. This class seems to extend the functionality of Wandb's default behavior by adding custom hooks for various training-related events, such as the beginning and end of training, saving checkpoints, predicting model outputs, etc.

Here are some key points about this code:

1. **Extending Wandb**: The class extendswandb's default logging behavior by implementing custom hooks for different stages of the training process.
2. **Training Event Hooks**:
   - `on_train_begin`: This hook is triggered when the training process begins. It sets up the logging for the Wandb run and logs the metadata (e.g., parameters, model architecture) needed for future steps in the project.
   - `on_log`: This hook logs various metrics during the training process, such as loss values, accuracy ratings, and training speed measurements.
   - `on_save`: Similarly, this log checkpoints during the training phase. 
3. **Custom Model Logging**:
   - The class includes methods for saving different versions of models (e.g., full model, final model) based on their performance and other metrics. These are logged in a particular format using logs artifacts.
4. **Pretraining Hyperparameter Search**:
   - When training with hyperparameters, this class finishes searching the hyperparameters space when a global minimum is achieved.

Here's an example use case for this class:

```python
# Create an instance of the Wandb logging class
wandb_logger = WandbLogger(
    log_model=not args.load_best_model_at_end,
    log_metrics=True,
    model_log_config={
        'log_model': True,
    }
)

# In your main training loop:
train_step(args, state, control)
```

In this context `args` is an instance of a configuration object which holds parameters of the dataset and model. The `state` argument should be initialized using `Trainer.begin()` to define other metadata onwandb.log(). 

This example demonstrates how to use the WandbLogger class in combination with the Trainer's training process for logging and analyzing machine learning experiments.


key: chroma:document
string_value:
CODE:

class CometCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that sends the logs to [Comet ML](https://www.comet.com/site/).
    """

    def __init__(self):
        if _is_comet_installed is False or _is_comet_recent_enough is False:
            raise RuntimeError(
                f"CometCallback requires comet-ml>={_MIN_COMET_VERSION} to be installed. Run `pip install comet-ml>={_MIN_COMET_VERSION}`."
            )
        self._initialized = False
        self._log_assets = False
        self._experiment = None

    def setup(self, args, state, model):
        """
        Setup the optional Comet integration.

        Environment:
        - **COMET_MODE** (`str`, *optional*, default to `get_or_create`):
            Control whether to create and log to a new Comet experiment or append to an existing experiment.
            It accepts the following values:
                * `get_or_create`: Decides automatically depending if
                  `COMET_EXPERIMENT_KEY` is set and whether an Experiment
                  with that key already exists or not.
                * `create`: Always create a new Comet Experiment.
                * `get`: Always try to append to an Existing Comet Experiment.
                  Requires `COMET_EXPERIMENT_KEY` to be set.
                * `ONLINE`: **deprecated**, used to create an online
                  Experiment. Use `COMET_START_ONLINE=1` instead.
                * `OFFLINE`: **deprecated**, used to created an offline
                  Experiment. Use `COMET_START_ONLINE=0` instead.
                * `DISABLED`: **deprecated**, used to disable Comet logging.
                  Use the `--report_to` flag to control the integrations used
                  for logging result instead.
        - **COMET_PROJECT_NAME** (`str`, *optional*):
            Comet project name for experiments.
        - **COMET_LOG_ASSETS** (`str`, *optional*, defaults to `TRUE`):
            Whether or not to log training assets (tf event logs, checkpoints, etc), to Comet. Can be `TRUE`, or
            `FALSE`.

        For a number of configurable items in the environment, see
        [here](https://www.comet.com/docs/v2/guides/experiment-management/configure-sdk/#explore-comet-configuration-options).
        """
        self._initialized = True
        log_assets = os.getenv("COMET_LOG_ASSETS", "FALSE").upper()
        if log_assets in {"TRUE", "1"}:
            self._log_assets = True
        if state.is_world_process_zero:
            comet_old_mode = os.getenv("COMET_MODE")

            mode = None
            online = None

            if comet_old_mode is not None:
                comet_old_mode = comet_old_mode.lower()

                if comet_old_mode == "online":
                    online = True
                elif comet_old_mode == "offline":
                    online = False
                elif comet_old_mode in ("get", "get_or_create", "create"):
                    mode = comet_old_mode
                elif comet_old_mode:
                    logger.warning("Invalid COMET_MODE env value %r, Comet logging is disabled", comet_old_mode)
                    return

            # For HPO, we always create a new experiment for each trial
            if state.is_hyper_param_search:
                if mode is not None:
                    logger.warning(
                        "Hyperparameter Search is enabled, forcing the creation of new experimetns, COMET_MODE value %r  is ignored",
                        comet_old_mode,
                    )
                mode = "create"

            import comet_ml

            # Do not use the default run_name as the experiment name
            if args.run_name is not None and args.run_name != args.output_dir:
                experiment_config = comet_ml.ExperimentConfig(name=args.run_name)
            else:
                experiment_config = comet_ml.ExperimentConfig()

            self._experiment = comet_ml.start(online=online, mode=mode, experiment_config=experiment_config)
            self._experiment.__internal_api__set_model_graph__(model, framework="transformers")

            params = {"args": args.to_dict()}

            if hasattr(model, "config") and model.config is not None:
                model_config = model.config.to_dict()
                params["config"] = model_config
            if hasattr(model, "peft_config") and model.peft_config is not None:
                peft_config = model.peft_config
                params["peft_config"] = peft_config

            self._experiment.__internal_api__log_parameters__(
                params, framework="transformers", source="manual", flatten_nested=True
            )

            if state.is_hyper_param_search:
                optimization_id = getattr(state, "trial_name", None)
                optimization_params = getattr(state, "trial_params", None)

                self._experiment.log_optimization(optimization_id=optimization_id, parameters=optimization_params)

    def on_train_begin(self, args, state, control, model=None, **kwargs):
        if not self._initialized:
            self.setup(args, state, model)

    def on_log(self, args, state, control, model=None, logs=None, **kwargs):
        if not self._initialized:
            self.setup(args, state, model)
        if state.is_world_process_zero:
            if self._experiment is not None:
                rewritten_logs = rewrite_logs(logs)
                self._experiment.__internal_api__log_metrics__(
                    rewritten_logs, step=state.global_step, epoch=state.epoch, framework="transformers"
                )

    def on_train_end(self, args, state, control, **kwargs):
        if self._initialized and state.is_world_process_zero:
            if self._experiment is not None:
                if self._log_assets is True:
                    logger.info("Logging checkpoints. This may take time.")
                    self._experiment.log_asset_folder(
                        args.output_dir, recursive=True, log_file_name=True, step=state.global_step
                    )

            # We create one experiment per trial in HPO mode
            if state.is_hyper_param_search:
                self._experiment.clean()
                self._initialized = False

    def on_predict(self, args, state, control, metrics, **kwargs):
        if not self._initialized:
            self.setup(args, state, model=None)
        if state.is_world_process_zero and self._experiment is not None:
            rewritten_metrics = rewrite_logs(metrics)
            self._experiment.__internal_api__log_metrics__(
                rewritten_metrics, step=state.global_step, epoch=state.epoch, framework="transformers"


SUMMARY:
**CometCallback Class Summary**

### Main Purpose and Functionality
The `CometCallback` class is a custom callback for the Trainer class that integrates with Comet ML to send logs, metrics, and experiment data.

### Key Implementation Details

* Sets up Comet integration based on environment variables (`COMET_MODE`, `COMET_PROJECT_NAME`, and `COMET_LOG_ASSETS`)
* Creates or appends to an existing Comet experiment
* Logs training assets (tf event logs, checkpoints, etc)
* Logs metrics and parameters during training and prediction phases
* Supports hyperparameter search with trial metadata

### Dependencies and Requirements

* comet-ml > `_MIN_COMET_VERSION`
* transformers framework

### Notable Interfaces and Methods

* `comet_ml.start()`: Creates or appends to a Comet experiment
* `comet_ml.__internal_api__log_parameters()`: Logs model parameters
* `comet_ml.__internal_api__log_metrics()`: Logs metrics during training and prediction phases


key: chroma:document
string_value:
CODE:

class AzureMLCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that sends the logs to [AzureML](https://pypi.org/project/azureml-sdk/).
    """

    def __init__(self, azureml_run=None):
        if not is_azureml_available():
            raise RuntimeError("AzureMLCallback requires azureml to be installed. Run `pip install azureml-sdk`.")
        self.azureml_run = azureml_run

    def on_init_end(self, args, state, control, **kwargs):
        from azureml.core.run import Run

        if self.azureml_run is None and state.is_world_process_zero:
            self.azureml_run = Run.get_context()

    def on_log(self, args, state, control, logs=None, **kwargs):
        if self.azureml_run and state.is_world_process_zero:
            for k, v in logs.items():
                if isinstance(v, (int, float)):


SUMMARY:
**AzureMLCallback Class Summary**

### Main Purpose and Functionality
The `AzureMLCallback` class serves as a plugin for the [`TrainerCallback`](https://...) class in machine learning environments. Its primary purpose is to send logs from a training process to Azure Machine Learning (AML) platform.

### Key Implementation Details

*   Initiates an AML run using the provided configuration (`azureml_run`) only if not already instantiated.
*   Logs numerical values from the `logs` dictionary with corresponding descriptions in the Azure ML database when running on the zeroth worker process (`state.is_world_process_zero`).

### Notable Dependencies and Requirements

*   The class requires the Azure Machine Learning SDK to be installed (`azureml-sdk` package).
*   It inherits from [`TrainerCallback`](https://...) and uses external classes/functions from the `azureml.core.run` module.

```markdown
### Code with precise technical terms:

class AzureMLCallback(TrainerCallback):
    """
    A TrainerCallback class that integrates with Azure Machine Learning.
    """

    def __init__(self, azureml_run=None):
        if not is_azureml_available():
            raise RuntimeError("AzureMLCallback requires azureml to be installed. Run `pip install azureml-sdk`.")
        self.azureml_run = azureml_run

    def on_init_end(self, args, state, control, **kwargs):
        Run.get_context()

    def on_log(self, args, state, control, logs=None, **kwargs):
        if selfazureml_run and state.is_world_process_zero:
            for k, v in logs.items():
                if isinstance(v, (int, float)):
                    selfazureml_run.log(k, v, description=k)
```


key: chroma:document
string_value:
CODE:

class MLflowCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that sends the logs to [MLflow](https://www.mlflow.org/). Can be disabled by setting
    environment variable `DISABLE_MLFLOW_INTEGRATION = TRUE`.
    """

    def __init__(self):
        if not is_mlflow_available():
            raise RuntimeError("MLflowCallback requires mlflow to be installed. Run `pip install mlflow`.")
        import mlflow

        self._MAX_PARAM_VAL_LENGTH = mlflow.utils.validation.MAX_PARAM_VAL_LENGTH
        self._MAX_PARAMS_TAGS_PER_BATCH = mlflow.utils.validation.MAX_PARAMS_TAGS_PER_BATCH

        self._initialized = False
        self._auto_end_run = False
        self._log_artifacts = False
        self._ml_flow = mlflow

    def setup(self, args, state, model):
        """
        Setup the optional MLflow integration.

        Environment:
        - **HF_MLFLOW_LOG_ARTIFACTS** (`str`, *optional*):
            Whether to use MLflow `.log_artifact()` facility to log artifacts. This only makes sense if logging to a
            remote server, e.g. s3 or GCS. If set to `True` or *1*, will copy each saved checkpoint on each save in
            [`TrainingArguments`]'s `output_dir` to the local or remote artifact storage. Using it without a remote
            storage will just copy the files to your artifact location.
        - **MLFLOW_TRACKING_URI** (`str`, *optional*):
            Whether to store runs at a specific path or remote server. Unset by default, which skips setting the
            tracking URI entirely.
        - **MLFLOW_EXPERIMENT_NAME** (`str`, *optional*, defaults to `None`):
            Whether to use an MLflow experiment_name under which to launch the run. Default to `None` which will point
            to the `Default` experiment in MLflow. Otherwise, it is a case sensitive name of the experiment to be
            activated. If an experiment with this name does not exist, a new experiment with this name is created.
        - **MLFLOW_TAGS** (`str`, *optional*):
            A string dump of a dictionary of key/value pair to be added to the MLflow run as tags. Example:
            `os.environ['MLFLOW_TAGS']='{"release.candidate": "RC1", "release.version": "2.2.0"}'`.
        - **MLFLOW_NESTED_RUN** (`str`, *optional*):
            Whether to use MLflow nested runs. If set to `True` or *1*, will create a nested run inside the current
            run.
        - **MLFLOW_RUN_ID** (`str`, *optional*):
            Allow to reattach to an existing run which can be usefull when resuming training from a checkpoint. When
            `MLFLOW_RUN_ID` environment variable is set, `start_run` attempts to resume a run with the specified run ID
            and other parameters are ignored.
        - **MLFLOW_FLATTEN_PARAMS** (`str`, *optional*, defaults to `False`):
            Whether to flatten the parameters dictionary before logging.
        - **MLFLOW_MAX_LOG_PARAMS** (`int`, *optional*):
            Set the maximum number of parameters to log in the run.
        """
        self._log_artifacts = os.getenv("HF_MLFLOW_LOG_ARTIFACTS", "FALSE").upper() in ENV_VARS_TRUE_VALUES
        self._nested_run = os.getenv("MLFLOW_NESTED_RUN", "FALSE").upper() in ENV_VARS_TRUE_VALUES
        self._tracking_uri = os.getenv("MLFLOW_TRACKING_URI", None)
        self._experiment_name = os.getenv("MLFLOW_EXPERIMENT_NAME", None)
        self._flatten_params = os.getenv("MLFLOW_FLATTEN_PARAMS", "FALSE").upper() in ENV_VARS_TRUE_VALUES
        self._run_id = os.getenv("MLFLOW_RUN_ID", None)
        self._max_log_params = os.getenv("MLFLOW_MAX_LOG_PARAMS", None)

        # "synchronous" flag is only available with mlflow version >= 2.8.0
        # https://github.com/mlflow/mlflow/pull/9705
        # https://github.com/mlflow/mlflow/releases/tag/v2.8.0
        self._async_log = packaging.version.parse(self._ml_flow.__version__) >= packaging.version.parse("2.8.0")

        logger.debug(
            f"MLflow experiment_name={self._experiment_name}, run_name={args.run_name}, nested={self._nested_run},"
            f" tracking_uri={self._tracking_uri}"
        )
        if state.is_world_process_zero:
            if not self._ml_flow.is_tracking_uri_set():
                if self._tracking_uri:
                    self._ml_flow.set_tracking_uri(self._tracking_uri)
                    logger.debug(f"MLflow tracking URI is set to {self._tracking_uri}")
                else:
                    logger.debug(
                        "Environment variable `MLFLOW_TRACKING_URI` is not provided and therefore will not be"
                        " explicitly set."
                    )
            else:
                logger.debug(f"MLflow tracking URI is set to {self._ml_flow.get_tracking_uri()}")

            if self._ml_flow.active_run() is None or self._nested_run or self._run_id:
                if self._experiment_name:
                    # Use of set_experiment() ensure that Experiment is created if not exists
                    self._ml_flow.set_experiment(self._experiment_name)
                self._ml_flow.start_run(run_name=args.run_name, nested=self._nested_run)
                logger.debug(f"MLflow run started with run_id={self._ml_flow.active_run().info.run_id}")
                self._auto_end_run = True
            combined_dict = args.to_dict()
            if hasattr(model, "config") and model.config is not None:
                model_config = model.config.to_dict()
                combined_dict = {**model_config, **combined_dict}
            combined_dict = flatten_dict(combined_dict) if self._flatten_params else combined_dict
            # remove params that are too long for MLflow
            for name, value in list(combined_dict.items()):
                # internally, all values are converted to str in MLflow
                if len(str(value)) > self._MAX_PARAM_VAL_LENGTH:
                    logger.warning(
                        f'Trainer is attempting to log a value of "{value}" for key "{name}" as a parameter. MLflow\'s'
                        " log_param() only accepts values no longer than 250 characters so we dropped this attribute."
                        " You can use `MLFLOW_FLATTEN_PARAMS` environment variable to flatten the parameters and"
                        " avoid this message."
                    )
                    del combined_dict[name]
            # MLflow cannot log more than 100 values in one go, so we have to split it
            combined_dict_items = list(combined_dict.items())
            if self._max_log_params and self._max_log_params.isdigit():
                max_log_params = int(self._max_log_params)
                if max_log_params < len(combined_dict_items):
                    logger.debug(
                        f"Reducing the number of parameters to log from {len(combined_dict_items)} to {max_log_params}."
                    )
                    combined_dict_items = combined_dict_items[:max_log_params]
            for i in range(0, len(combined_dict_items), self._MAX_PARAMS_TAGS_PER_BATCH):
                if self._async_log:
                    self._ml_flow.log_params(
                        dict(combined_dict_items[i : i + self._MAX_PARAMS_TAGS_PER_BATCH]), synchronous=False
                    )
                else:
                    self._ml_flow.log_params(dict(combined_dict_items[i : i + self._MAX_PARAMS_TAGS_PER_BATCH]))
            mlflow_tags = os.getenv("MLFLOW_TAGS", None)
            if mlflow_tags:
                mlflow_tags = json.loads(mlflow_tags)
                self._ml_flow.set_tags(mlflow_tags)
        self._initialized = True

    def on_train_begin(self, args, state, control, model=None, **kwargs):
        if not self._initialized:
            self.setup(args, state, model)

    def on_log(self, args, state, control, logs, model=None, **kwargs):
        if not self._initialized:
            self.setup(args, state, model)
        if state.is_world_process_zero:
            metrics = {}
            for k, v in logs.items():
                if isinstance(v, (int, float)):
                    metrics[k] = v
                elif isinstance(v, torch.Tensor) and v.numel() == 1:
                    metrics[k] = v.item()
                else:
                    logger.warning(
                        f'Trainer is attempting to log a value of "{v}" of type {type(v)} for key "{k}" as a metric. '
                        "MLflow's log_metric() only accepts float and int types so we dropped this attribute."
                    )

            if self._async_log:
                self._ml_flow.log_metrics(metrics=metrics, step=state.global_step, synchronous=False)
            else:
                self._ml_flow.log_metrics(metrics=metrics, step=state.global_step)

    def on_train_end(self, args, state, control, **kwargs):
        if self._initialized and state.is_world_process_zero:
            if self._auto_end_run and self._ml_flow.active_run():
                self._ml_flow.end_run()

    def on_save(self, args, state, control, **kwargs):
        if self._initialized and state.is_world_process_zero and self._log_artifacts:
            ckpt_dir = f"checkpoint-{state.global_step}"
            artifact_path = os.path.join(args.output_dir, ckpt_dir)
            logger.info(f"Logging checkpoint artifacts in {ckpt_dir}. This may take time.")
            self._ml_flow.pyfunc.log_model(
                ckpt_dir,
                artifacts={"model_path": artifact_path},
                python_model=self._ml_flow.pyfunc.PythonModel(),
            )

    def __del__(self):
        # if the previous run is not terminated correctly, the fluent API will
        # not let you start a new run before the previous one is killed
        if (
            self._auto_end_run
            and callable(getattr(self._ml_flow, "active_run", None))
            and self._ml_flow.active_run() is not None
        ):


SUMMARY:
This code appears to be part of a deep learning framework's Trainer class. It extends the Trainer class from PyTorch Lightning, providing additional functionality for logging parameters and metrics using the MLflow library.

Key Features:

1. **Initialization**: The `setup` method is called when the trainer starts, which initializes the MLflow session and logs the model configuration.
2. **Logging Parameters**: The `log_params` method is used to log parameters, with features like asynchronous execution and batch splitting to avoid exceeding MLflow's parameter limit (100).
3. **Async Logging**: The `_async_log` flag enables asynchronous logging, allowing for faster training times.
4. **Metrics Logging**: The `on_log` method logs metrics during training, using the same asynchronous mode as parameters logging.
5. **Auto-Ending Runs**: If `auto_end_run` is True and a previous run is not terminated correctly, the trainer manually ends the current run to avoid issues with starting a new one.

This code aims to provide a smooth experience for MLflow-based PyTorch Lightning training, allowing users to easily log their models, metrics, and parameters while working within the Python library.


key: chroma:document
string_value:
CODE:

class DagsHubCallback(MLflowCallback):
    """
    A [`TrainerCallback`] that logs to [DagsHub](https://dagshub.com/). Extends [`MLflowCallback`]
    """

    def __init__(self):
        super().__init__()
        if not is_dagshub_available():
            raise ImportError("DagsHubCallback requires dagshub to be installed. Run `pip install dagshub`.")

        from dagshub.upload import Repo

        self.Repo = Repo

    def setup(self, *args, **kwargs):
        """
        Setup the DagsHub's Logging integration.

        Environment:
        - **HF_DAGSHUB_LOG_ARTIFACTS** (`str`, *optional*):
                Whether to save the data and model artifacts for the experiment. Default to `False`.
        """

        self.log_artifacts = os.getenv("HF_DAGSHUB_LOG_ARTIFACTS", "FALSE").upper() in ENV_VARS_TRUE_VALUES
        self.name = os.getenv("HF_DAGSHUB_MODEL_NAME") or "main"
        self.remote = os.getenv("MLFLOW_TRACKING_URI")
        self.repo = self.Repo(
            owner=self.remote.split(os.sep)[-2],
            name=self.remote.split(os.sep)[-1].split(".")[0],
            branch=os.getenv("BRANCH") or "main",
        )
        self.path = Path("artifacts")

        if self.remote is None:
            raise RuntimeError(
                "DagsHubCallback requires the `MLFLOW_TRACKING_URI` environment variable to be set. Did you run"
                " `dagshub.init()`?"
            )

        super().setup(*args, **kwargs)

    def on_train_end(self, args, state, control, **kwargs):
        if self.log_artifacts:
            if getattr(self, "train_dataloader", None):
                torch.save(self.train_dataloader.dataset, os.path.join(args.output_dir, "dataset.pt"))



SUMMARY:
**Class Summary: DagsHubCallback**

**Purpose and Functionality**
The `DagsHubCallback` class is an extension of[`MLflowCallback`] that integrates with [DagsHub](https://dagshub.com/). It logs experiments to DagsHub, allowing for versioning and collaboration of machine learning workflows.

**Key Implementation Details**

* Sets up logging integration with DagsHub using environment variables `HF_DAGSHUB_LOG_ARTIFACTS`, `HF_DAGSHUB_MODEL_NAME`, and `MLFLOW_TRACKING_URI`.
* Stores experiment artifacts in a designated directory (`artifacts`) after training ends.
* Integrates with the `Repo` class from the `dagshub.upload` module.

**Notable Dependencies or Requirements**

* `MLflowCallback`
* `DagShub` (installed via pip)
* Import of `Repo` from `dagshub_upload`

Note: This summary focuses on the main purpose, key implementation details, and notable dependencies. It preserves class/function/variable names exactly and provides a concise overview of the code's functionality.


key: chroma:document
string_value:
CODE:

class NeptuneMissingConfiguration(Exception):
    def __init__(self):
        super().__init__(
            """
        ------ Unsupported ---- We were not able to create new runs. You provided a custom Neptune run to
        `NeptuneCallback` with the `run` argument. For the integration to work fully, provide your `api_token` and
        `project` by saving them as environment variables or passing them to the callback.
        """


SUMMARY:
**Summary:**

The `NeptuneMissingConfiguration` class represents an exception that occurs when a custom Neptune run is provided without required api_token and project information.

**Key Points:**

* **Primary Purpose:** Raised when a custom Neptune run lacks essential configuration details (api_token and project).
* **Implementation Details:**
	+ Utilizes the parent `Exception` class.
	+ Uses a multiline string in its constructor to display an error message.
* **Notable Dependencies/Requirements:**
	+ The `NeptuneCallback` integration, which depends on `api_token` and `project`.
	+ Environment variable or callback passing mechanism for config data.


key: chroma:document
string_value:
CODE:

class NeptuneCallback(TrainerCallback):
    """TrainerCallback that sends the logs to [Neptune](https://app.neptune.ai).

    Args:
        api_token (`str`, *optional*): Neptune API token obtained upon registration.
            You can leave this argument out if you have saved your token to the `NEPTUNE_API_TOKEN` environment
            variable (strongly recommended). See full setup instructions in the
            [docs](https://docs.neptune.ai/setup/installation).
        project (`str`, *optional*): Name of an existing Neptune project, in the form "workspace-name/project-name".
            You can find and copy the name in Neptune from the project settings -> Properties. If None (default), the
            value of the `NEPTUNE_PROJECT` environment variable is used.
        name (`str`, *optional*): Custom name for the run.
        base_namespace (`str`, *optional*, defaults to "finetuning"): In the Neptune run, the root namespace
            that will contain all of the metadata logged by the callback.
        log_parameters (`bool`, *optional*, defaults to `True`):
            If True, logs all Trainer arguments and model parameters provided by the Trainer.
        log_checkpoints (`str`, *optional*): If "same", uploads checkpoints whenever they are saved by the Trainer.
            If "last", uploads only the most recently saved checkpoint. If "best", uploads the best checkpoint (among
            the ones saved by the Trainer). If `None`, does not upload checkpoints.
        run (`Run`, *optional*): Pass a Neptune run object if you want to continue logging to an existing run.
            Read more about resuming runs in the [docs](https://docs.neptune.ai/logging/to_existing_object).
        **neptune_run_kwargs (*optional*):
            Additional keyword arguments to be passed directly to the
            [`neptune.init_run()`](https://docs.neptune.ai/api/neptune#init_run) function when a new run is created.

    For instructions and examples, see the [Transformers integration
    guide](https://docs.neptune.ai/integrations/transformers) in the Neptune documentation.
    """

    integration_version_key = "source_code/integrations/transformers"
    model_parameters_key = "model_parameters"
    trial_name_key = "trial"
    trial_params_key = "trial_params"
    trainer_parameters_key = "trainer_parameters"
    flat_metrics = {"train/epoch"}

    def __init__(
        self,
        *,
        api_token: Optional[str] = None,
        project: Optional[str] = None,
        name: Optional[str] = None,
        base_namespace: str = "finetuning",
        run=None,
        log_parameters: bool = True,
        log_checkpoints: Optional[str] = None,
        **neptune_run_kwargs,
    ):
        if not is_neptune_available():
            raise ValueError(
                "NeptuneCallback requires the Neptune client library to be installed. "
                "To install the library, run `pip install neptune`."
            )

        try:
            from neptune import Run
            from neptune.internal.utils import verify_type
        except ImportError:
            from neptune.new.internal.utils import verify_type
            from neptune.new.metadata_containers.run import Run

        verify_type("api_token", api_token, (str, type(None)))
        verify_type("project", project, (str, type(None)))
        verify_type("name", name, (str, type(None)))
        verify_type("base_namespace", base_namespace, str)
        verify_type("run", run, (Run, type(None)))
        verify_type("log_parameters", log_parameters, bool)
        verify_type("log_checkpoints", log_checkpoints, (str, type(None)))

        self._base_namespace_path = base_namespace
        self._log_parameters = log_parameters
        self._log_checkpoints = log_checkpoints
        self._initial_run: Optional[Run] = run

        self._run = None
        self._is_monitoring_run = False
        self._run_id = None
        self._force_reset_monitoring_run = False
        self._init_run_kwargs = {"api_token": api_token, "project": project, "name": name, **neptune_run_kwargs}

        self._volatile_checkpoints_dir = None
        self._should_upload_checkpoint = self._log_checkpoints is not None
        self._recent_checkpoint_path = None

        if self._log_checkpoints in {"last", "best"}:
            self._target_checkpoints_namespace = f"checkpoints/{self._log_checkpoints}"
            self._should_clean_recently_uploaded_checkpoint = True
        else:
            self._target_checkpoints_namespace = "checkpoints"
            self._should_clean_recently_uploaded_checkpoint = False

    def _stop_run_if_exists(self):
        if self._run:
            self._run.stop()
            del self._run
            self._run = None

    def _initialize_run(self, **additional_neptune_kwargs):
        try:
            from neptune import init_run
            from neptune.exceptions import NeptuneMissingApiTokenException, NeptuneMissingProjectNameException
        except ImportError:
            from neptune.new import init_run
            from neptune.new.exceptions import NeptuneMissingApiTokenException, NeptuneMissingProjectNameException

        self._stop_run_if_exists()

        try:
            run_params = additional_neptune_kwargs.copy()
            run_params.update(self._init_run_kwargs)
            self._run = init_run(**run_params)
            self._run_id = self._run["sys/id"].fetch()
        except (NeptuneMissingProjectNameException, NeptuneMissingApiTokenException) as e:
            raise NeptuneMissingConfiguration() from e

    def _use_initial_run(self):
        self._run = self._initial_run
        self._is_monitoring_run = True
        self._run_id = self._run["sys/id"].fetch()
        self._initial_run = None

    def _ensure_run_with_monitoring(self):
        if self._initial_run is not None:
            self._use_initial_run()
        else:
            if not self._force_reset_monitoring_run and self._is_monitoring_run:
                return

            if self._run and not self._is_monitoring_run and not self._force_reset_monitoring_run:
                self._initialize_run(with_id=self._run_id)
                self._is_monitoring_run = True
            else:
                self._initialize_run()
                self._force_reset_monitoring_run = False

    def _ensure_at_least_run_without_monitoring(self):
        if self._initial_run is not None:
            self._use_initial_run()
        else:
            if not self._run:
                self._initialize_run(
                    with_id=self._run_id,
                    capture_stdout=False,
                    capture_stderr=False,
                    capture_hardware_metrics=False,
                    capture_traceback=False,
                )
                self._is_monitoring_run = False

    @property
    def run(self):
        if self._run is None:
            self._ensure_at_least_run_without_monitoring()
        return self._run

    @property
    def _metadata_namespace(self):
        return self.run[self._base_namespace_path]

    def _log_integration_version(self):
        self.run[NeptuneCallback.integration_version_key] = version

    def _log_trainer_parameters(self, args):
        self._metadata_namespace[NeptuneCallback.trainer_parameters_key] = args.to_sanitized_dict()

    def _log_model_parameters(self, model):
        from neptune.utils import stringify_unsupported

        if model and hasattr(model, "config") and model.config is not None:
            self._metadata_namespace[NeptuneCallback.model_parameters_key] = stringify_unsupported(
                model.config.to_dict()
            )

    def _log_hyper_param_search_parameters(self, state):
        if state and hasattr(state, "trial_name"):
            self._metadata_namespace[NeptuneCallback.trial_name_key] = state.trial_name

        if state and hasattr(state, "trial_params") and state.trial_params is not None:
            self._metadata_namespace[NeptuneCallback.trial_params_key] = state.trial_params

    def _log_model_checkpoint(self, source_directory: str, checkpoint: str):
        target_path = relative_path = os.path.join(source_directory, checkpoint)

        if self._volatile_checkpoints_dir is not None:
            consistent_checkpoint_path = os.path.join(self._volatile_checkpoints_dir, checkpoint)
            try:
                # Remove leading ../ from a relative path.
                cpkt_path = relative_path.replace("..", "").lstrip(os.path.sep)
                copy_path = os.path.join(consistent_checkpoint_path, cpkt_path)
                shutil.copytree(relative_path, copy_path)
                target_path = consistent_checkpoint_path
            except IOError as e:
                logger.warning(
                    "NeptuneCallback was unable to made a copy of checkpoint due to I/O exception: '{}'. "
                    "Could fail trying to upload.".format(e)
                )

        self._metadata_namespace[self._target_checkpoints_namespace].upload_files(target_path)

        if self._should_clean_recently_uploaded_checkpoint and self._recent_checkpoint_path is not None:
            self._metadata_namespace[self._target_checkpoints_namespace].delete_files(self._recent_checkpoint_path)

        self._recent_checkpoint_path = relative_path

    def on_init_end(self, args, state, control, **kwargs):
        self._volatile_checkpoints_dir = None
        if self._log_checkpoints and (args.overwrite_output_dir or args.save_total_limit is not None):
            self._volatile_checkpoints_dir = tempfile.TemporaryDirectory().name

        if self._log_checkpoints == "best" and not args.load_best_model_at_end:
            raise ValueError("To save the best model checkpoint, the load_best_model_at_end argument must be enabled.")

    def on_train_begin(self, args, state, control, model=None, **kwargs):
        if not state.is_world_process_zero:
            return

        self._ensure_run_with_monitoring()
        self._force_reset_monitoring_run = True

        self._log_integration_version()
        if self._log_parameters:
            self._log_trainer_parameters(args)
            self._log_model_parameters(model)

        if state.is_hyper_param_search:
            self._log_hyper_param_search_parameters(state)

    def on_train_end(self, args, state, control, **kwargs):
        self._stop_run_if_exists()

    def __del__(self):
        if self._volatile_checkpoints_dir is not None:
            shutil.rmtree(self._volatile_checkpoints_dir, ignore_errors=True)

        self._stop_run_if_exists()

    def on_save(self, args, state, control, **kwargs):
        if self._should_upload_checkpoint:
            self._log_model_checkpoint(args.output_dir, f"checkpoint-{state.global_step}")

    def on_evaluate(self, args, state, control, metrics=None, **kwargs):
        if self._log_checkpoints == "best":
            best_metric_name = args.metric_for_best_model
            if not best_metric_name.startswith("eval_"):
                best_metric_name = f"eval_{best_metric_name}"

            metric_value = metrics.get(best_metric_name)

            operator = np.greater if args.greater_is_better else np.less

            self._should_upload_checkpoint = state.best_metric is None or operator(metric_value, state.best_metric)

    @classmethod
    def get_run(cls, trainer):
        for callback in trainer.callback_handler.callbacks:
            if isinstance(callback, cls):
                return callback.run

        raise Exception("The trainer doesn't have a NeptuneCallback configured.")

    def on_log(self, args, state, control, logs: Optional[Dict[str, float]] = None, **kwargs):
        if not state.is_world_process_zero:
            return

        if logs is not None:
            for name, value in rewrite_logs(logs).items():
                if isinstance(value, (int, float)):
                    if name in NeptuneCallback.flat_metrics:
                        self._metadata_namespace[name] = value
                    else:


SUMMARY:
Here is a succinct summary of the provided code:

**Neptune Callback Class**

This class is part of a larger framework that integrates with Neptune.io for tracking machine learning experiments. It inherits from PyTorch's `Callback` class.

**Functions and Methods**

*   `on_init_end`: Called after the initial execution phase; creates temporary directory if desired checkpoints are enabled.
*   `on_train_begin`, `on_train_end`, `on_save`, `on_evaluate`, and `on_log`: Hooks for integrating with Neptune.io; tracks experiment data, including parameters, model hyperparameters, training metrics, and checkpoint uploads.
*   `_log_trainer_parameters` and `_log_model_parameters`: Logs trainer and model parameters to the experiment dashboard.
*   `get_run(self, trainer)`: Returns the Neptune callback instance associated with a specified trainer or raises an exception if not found.

**Log Rewrite**

In `on_log`, logs are rewritten by replacing non-finite values with NaN (Not a Number). The logging function uses the `step` value as part of its arguments to maintain accurate timestamp representation.


key: chroma:document
string_value:
CODE:

class CodeCarbonCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that tracks the CO2 emission of training.
    """

    def __init__(self):
        if not is_codecarbon_available():
            raise RuntimeError(
                "CodeCarbonCallback requires `codecarbon` to be installed. Run `pip install codecarbon`."
            )
        elif torch.version.hip:
            raise RuntimeError(
                "CodeCarbonCallback requires `codecarbon` package, which is not compatible with AMD ROCm (https://github.com/mlco2/codecarbon/pull/490). When using the Trainer, please specify the `report_to` argument (https://huggingface.co/docs/transformers/v4.39.3/en/main_classes/trainer#transformers.TrainingArguments.report_to) to disable CodeCarbonCallback."
            )

        import codecarbon

        self._codecarbon = codecarbon
        self.tracker = None

    def on_init_end(self, args, state, control, **kwargs):
        if self.tracker is None and state.is_local_process_zero:
            # CodeCarbon will automatically handle environment variables for configuration
            self.tracker = self._codecarbon.EmissionsTracker(output_dir=args.output_dir)

    def on_train_begin(self, args, state, control, model=None, **kwargs):
        if self.tracker and state.is_local_process_zero:
            self.tracker.start()

    def on_train_end(self, args, state, control, **kwargs):
        if self.tracker and state.is_local_process_zero:


SUMMARY:
**Class Summary: CodeCarbonCallback**

*   **Purpose**: Tracks CO2 emission of training through integration with `codecarbon`.
*   **Functionality**: Provides emissions tracking during training, utilizes environment variables for configuration.

## Implementation Details

### Initializes Callback

-   Verifies `codecarbon` installation availability.
-   Raises errors for incompatible ROCm versions or missing installation.
-   Imports and initializes `codecarbon` package.

### Tracker Initialization

-   Checks if tracker is None and local process zero.
-   Sets up tracking configuration for CO2 emissions, utilizing environment variables.

### Training Life Cycle Methods

*   `on_init_end`: Initializes trackers when training begins, ensuring setup of emission monitoring.
*   `on_train_begin` and `on_train_end`: Starts and stops trackers during train events.

## Dependencies

-   Related Classes: `TrainerCallback`, `codecarbon.EmissionsTracker`
-   External Dependence: `codecarbon` package (with installation via pip)


key: chroma:document
string_value:
CODE:

class ClearMLCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that sends the logs to [ClearML](https://clear.ml/).

    Environment:
    - **CLEARML_PROJECT** (`str`, *optional*, defaults to `HuggingFace Transformers`):
        ClearML project name.
    - **CLEARML_TASK** (`str`, *optional*, defaults to `Trainer`):
        ClearML task name.
    - **CLEARML_LOG_MODEL** (`bool`, *optional*, defaults to `False`):
        Whether to log models as artifacts during training.
    """

    log_suffix = ""

    _hparams_section = "Transformers"
    _model_config_section = "Model Configuration"
    _ignore_hparams_overrides = "_ignore_hparams_ui_overrides_"
    _ignoge_model_config_overrides = "_ignore_model_config_ui_overrides_"
    _model_config_description = "The configuration of model number {}."
    _model_config_description_note = (
        "Note that, when cloning this task and running it remotely,"
        " the configuration might be applied to another model instead of this one."
        " To avoid this, initialize the task externally by calling `Task.init`"
        " before the `ClearMLCallback` is instantiated."
    )
    _train_run_counter = 0
    _model_connect_counter = 0
    _task_created_in_callback = False
    _should_close_on_train_end = None

    def __init__(self):
        if is_clearml_available():
            import clearml

            self._clearml = clearml
        else:
            raise RuntimeError("ClearMLCallback requires 'clearml' to be installed. Run `pip install clearml`.")

        self._initialized = False
        self._clearml_task = None

        self._log_model = False
        self._checkpoints_saved = []

    def setup(self, args, state, model, tokenizer, **kwargs):
        if self._clearml is None:
            return
        if self._initialized:
            return
        ClearMLCallback._train_run_counter += 1
        ClearMLCallback._model_connect_counter += 1
        ClearMLCallback.log_suffix = (
            "" if ClearMLCallback._train_run_counter == 1 else "_" + str(ClearMLCallback._train_run_counter)
        )
        if state.is_world_process_zero:
            logger.info("Automatic ClearML logging enabled.")
            if self._clearml_task is None:
                if ClearMLCallback._should_close_on_train_end is None:
                    if not self._clearml.Task.running_locally() or self._clearml.Task.current_task():
                        ClearMLCallback._should_close_on_train_end = False
                    else:
                        ClearMLCallback._should_close_on_train_end = True

                # This might happen when running inside of a pipeline, where the task is already initialized
                # from outside of Hugging Face
                if self._clearml.Task.running_locally() and self._clearml.Task.current_task():
                    self._clearml_task = self._clearml.Task.current_task()
                    self._log_model = os.getenv(
                        "CLEARML_LOG_MODEL",
                        "FALSE" if not ClearMLCallback._task_created_in_callback else "TRUE",
                    ).upper() in ENV_VARS_TRUE_VALUES.union({"TRUE"})
                    logger.info("External ClearML Task has been connected.")
                else:
                    self._clearml_task = self._clearml.Task.init(
                        project_name=os.getenv("CLEARML_PROJECT", "HuggingFace Transformers"),
                        task_name=os.getenv("CLEARML_TASK", "Trainer"),
                        auto_connect_frameworks={"tensorboard": False, "pytorch": False},
                        output_uri=True,
                    )
                    self._log_model = os.getenv("CLEARML_LOG_MODEL", "TRUE").upper() in ENV_VARS_TRUE_VALUES.union(
                        {"TRUE"}
                    )
                    ClearMLCallback._task_created_in_callback = True
                    logger.info("ClearML Task has been initialized.")
                self._initialized = True

            suffixed_hparams_section = ClearMLCallback._hparams_section + ClearMLCallback.log_suffix
            ignore_hparams_config_section = suffixed_hparams_section + "/" + ClearMLCallback._ignore_hparams_overrides
            if self._clearml.Task.running_locally():
                self._copy_training_args_as_hparams(args, suffixed_hparams_section)
                self._clearml_task.set_parameter(
                    name=ignore_hparams_config_section,
                    value=True,
                    value_type=bool,
                    description=(
                        "If True, ignore Transformers hyperparameters overrides done in the UI/backend "
                        + "when running remotely. Otherwise, the overrides will be applied when running remotely"
                    ),
                )
            elif not self._clearml_task.get_parameter(ignore_hparams_config_section, default=True, cast=True):
                self._clearml_task.connect(args, suffixed_hparams_section)
            else:
                self._copy_training_args_as_hparams(
                    args, ClearMLCallback._hparams_section + ClearMLCallback.log_suffix
                )

            if getattr(model, "config", None) is not None:
                ignore_model_config_section = (
                    suffixed_hparams_section + "/" + ClearMLCallback._ignoge_model_config_overrides
                )
                configuration_object_description = ClearMLCallback._model_config_description.format(
                    ClearMLCallback._model_connect_counter
                )
                if ClearMLCallback._model_connect_counter != ClearMLCallback._train_run_counter:
                    configuration_object_description += " " + ClearMLCallback._model_config_description_note
                if self._clearml.Task.running_locally():
                    self._clearml_task.set_parameter(
                        name=ignore_model_config_section,
                        value=True,
                        value_type=bool,
                        description=(
                            "If True, ignore Transformers model configuration overrides done in the UI/backend "
                            + "when running remotely. Otherwise, the overrides will be applied when running remotely"
                        ),
                    )
                    self._clearml_task.set_configuration_object(
                        name=ClearMLCallback._model_config_section + ClearMLCallback.log_suffix,
                        config_dict=model.config.to_dict(),
                        description=configuration_object_description,
                    )
                elif not self._clearml_task.get_parameter(ignore_model_config_section, default=True, cast=True):
                    model.config = model.config.from_dict(
                        self._clearml_task.get_configuration_object_as_dict(
                            ClearMLCallback._model_config_section + ClearMLCallback.log_suffix
                        )
                    )
                else:
                    self._clearml_task.set_configuration_object(
                        name=ClearMLCallback._model_config_section + ClearMLCallback.log_suffix,
                        config_dict=model.config.to_dict(),
                        description=configuration_object_description,
                    )

    def on_train_begin(self, args, state, control, model=None, tokenizer=None, **kwargs):
        if self._clearml is None:
            return
        self._checkpoints_saved = []
        if state.is_hyper_param_search:
            self._initialized = False
        if not self._initialized:
            self.setup(args, state, model, tokenizer, **kwargs)

    def on_train_end(self, args, state, control, **kwargs):
        if ClearMLCallback._should_close_on_train_end:
            self._clearml_task.close()
            ClearMLCallback._train_run_counter = 0

    def on_log(self, args, state, control, model=None, tokenizer=None, logs=None, **kwargs):
        if self._clearml is None:
            return
        if not self._initialized:
            self.setup(args, state, model, tokenizer, **kwargs)
        if state.is_world_process_zero:
            eval_prefix = "eval_"
            eval_prefix_len = len(eval_prefix)
            test_prefix = "test_"
            test_prefix_len = len(test_prefix)
            single_value_scalars = [
                "train_runtime",
                "train_samples_per_second",
                "train_steps_per_second",
                "train_loss",
                "total_flos",
                "epoch",
            ]
            for k, v in logs.items():
                if isinstance(v, (int, float)):
                    if k in single_value_scalars:
                        self._clearml_task.get_logger().report_single_value(
                            name=k + ClearMLCallback.log_suffix, value=v
                        )
                    elif k.startswith(eval_prefix):
                        self._clearml_task.get_logger().report_scalar(
                            title="eval" + ClearMLCallback.log_suffix,
                            series=k[eval_prefix_len:],
                            value=v,
                            iteration=state.global_step,
                        )
                    elif k.startswith(test_prefix):
                        self._clearml_task.get_logger().report_scalar(
                            title="test" + ClearMLCallback.log_suffix,
                            series=k[test_prefix_len:],
                            value=v,
                            iteration=state.global_step,
                        )
                    else:
                        self._clearml_task.get_logger().report_scalar(
                            title="train" + ClearMLCallback.log_suffix,
                            series=k,
                            value=v,
                            iteration=state.global_step,
                        )
                else:
                    logger.warning(
                        "Trainer is attempting to log a value of "
                        f'"{v}" of type {type(v)} for key "{k}" as a scalar. '
                        "This invocation of ClearML logger's  report_scalar() "
                        "is incorrect so we dropped this attribute."
                    )

    def on_save(self, args, state, control, **kwargs):
        if self._log_model and self._clearml_task and state.is_world_process_zero:
            ckpt_dir = f"checkpoint-{state.global_step}"
            artifact_path = os.path.join(args.output_dir, ckpt_dir)
            name = ckpt_dir + ClearMLCallback.log_suffix
            logger.info(f"Logging checkpoint artifact `{name}`. This may take some time.")
            output_model = self._clearml.OutputModel(task=self._clearml_task, name=name)
            output_model.connect(task=self._clearml_task, name=name)
            output_model.update_weights_package(
                weights_path=artifact_path,
                target_filename=ckpt_dir,
                iteration=state.global_step,
                auto_delete_file=False,
            )
            self._checkpoints_saved.append(output_model)
            while args.save_total_limit and args.save_total_limit < len(self._checkpoints_saved):
                try:
                    self._clearml.model.Model.remove(
                        self._checkpoints_saved[0],
                        delete_weights_file=True,
                        force=True,
                        raise_on_errors=True,
                    )
                except Exception as e:
                    logger.warning(
                        "Could not remove checkpoint `{}` after going over the `save_total_limit`. Error is: {}".format(
                            self._checkpoints_saved[0].name, e
                        )
                    )
                    break
                self._checkpoints_saved = self._checkpoints_saved[1:]

    def _copy_training_args_as_hparams(self, training_args, prefix):
        as_dict = {
            field.name: getattr(training_args, field.name)
            for field in fields(training_args)
            if field.init and not field.name.endswith("_token")
        }
        flat_dict = {str(k): v for k, v in self._clearml.utilities.proxy_object.flatten_dictionary(as_dict).items()}


SUMMARY:
This is a implementation of the `Trainer` class with additional features provided by the `ClearML` library for logging and saving model checkpoints. 

Here are some key points of this code:

- **Logging**: The code has several methods (`on_train_begin`, `on_train_end`, `on_log`) that are used to log various types of data (e.g., scalars, single values) to the ClearML library.

- **Checkpoint Saving**: When a model checkpoint is saved, it creates an output model using `ClearML` and saves its weights package. This process continues until the number of saved checkpoints reaches the specified limit.

- **Configuration**: The class takes care of initializing itself when necessary (i.e., in `on_train_begin`) and closes itself if this option has been configured (`on_train_end`).

Here is a concise summary:

**Purpose**: To provide additional functionality for logging data to ClearML and saving model checkpoints during training.

**Key Features**:
- Supports logging various types of data.
- Automatically saves model checkpoints using ClearML's OutputModel feature.
- Can be initialized or closed based on configuration.


key: chroma:document
string_value:
CODE:

class FlyteCallback(TrainerCallback):
    """A [`TrainerCallback`] that sends the logs to [Flyte](https://flyte.org/).
    NOTE: This callback only works within a Flyte task.

    Args:
        save_log_history (`bool`, *optional*, defaults to `True`):
            When set to True, the training logs are saved as a Flyte Deck.

        sync_checkpoints (`bool`, *optional*, defaults to `True`):
            When set to True, checkpoints are synced with Flyte and can be used to resume training in the case of an
            interruption.

    Example:

    ```python
    # Note: This example skips over some setup steps for brevity.
    from flytekit import current_context, task


    @task
    def train_hf_transformer():
        cp = current_context().checkpoint
        trainer = Trainer(..., callbacks=[FlyteCallback()])
        output = trainer.train(resume_from_checkpoint=cp.restore())
    ```
    """

    def __init__(self, save_log_history: bool = True, sync_checkpoints: bool = True):
        super().__init__()
        if not is_flytekit_available():
            raise ImportError("FlyteCallback requires flytekit to be installed. Run `pip install flytekit`.")

        if not is_flyte_deck_standard_available() or not is_pandas_available():
            logger.warning(
                "Syncing log history requires both flytekitplugins-deck-standard and pandas to be installed. "
                "Run `pip install flytekitplugins-deck-standard pandas` to enable this feature."
            )
            save_log_history = False

        from flytekit import current_context

        self.cp = current_context().checkpoint
        self.save_log_history = save_log_history
        self.sync_checkpoints = sync_checkpoints

    def on_save(self, args, state, control, **kwargs):
        if self.sync_checkpoints and state.is_world_process_zero:
            ckpt_dir = f"checkpoint-{state.global_step}"
            artifact_path = os.path.join(args.output_dir, ckpt_dir)

            logger.info(f"Syncing checkpoint in {ckpt_dir} to Flyte. This may take time.")
            self.cp.save(artifact_path)

    def on_train_end(self, args, state, control, **kwargs):
        if self.save_log_history:
            import pandas as pd
            from flytekit import Deck
            from flytekitplugins.deck.renderer import TableRenderer

            log_history_df = pd.DataFrame(state.log_history)


SUMMARY:
**Summary: FlyteCallback Class**

*   **Purpose**: Enforces communication with [Flyte](https://flyte.org/) task-level callback.
*   **Functionality**: 
    *   Synchronizes training checkpoints with Flyte.
    *   Saves training logs as a Flyte Deck.

**Key Implementation Details**

*   Initializes logging artifacts using `current_context().checkpoint`.
*   Includes dependencies `flytekit`, `pandas`, and `flytekitplugins.deck.renderer` (TableRenderer).
*   Uses `on_save` method to save checkpoints in sync with Flyte.

**Notable Requirements/Dependencies**

*   [Flyte](https://flyte.org/)
*   `is_flytekit_available()` check ensures proper flytekit installation.
*   `pandas` required for saving log history.
*   `TableRenderer` from `flytekitplugins.deck.renderer`.

This summary highlights the key purpose, functionality, and dependencies of the FlyteCallback class in a concise manner.


key: chroma:document
string_value:
CODE:

class DVCLiveCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that sends the logs to [DVCLive](https://www.dvc.org/doc/dvclive).

    Use the environment variables below in `setup` to configure the integration. To customize this callback beyond
    those environment variables, see [here](https://dvc.org/doc/dvclive/ml-frameworks/huggingface).

    Args:
        live (`dvclive.Live`, *optional*, defaults to `None`):
            Optional Live instance. If None, a new instance will be created using **kwargs.
        log_model (Union[Literal["all"], bool], *optional*, defaults to `None`):
            Whether to use `dvclive.Live.log_artifact()` to log checkpoints created by [`Trainer`]. If set to `True`,
            the final checkpoint is logged at the end of training. If set to `"all"`, the entire
            [`TrainingArguments`]'s `output_dir` is logged at each checkpoint.
    """

    def __init__(
        self,
        live: Optional[Any] = None,
        log_model: Optional[Union[Literal["all"], bool]] = None,
        **kwargs,
    ):
        if not is_dvclive_available():
            raise RuntimeError("DVCLiveCallback requires dvclive to be installed. Run `pip install dvclive`.")
        from dvclive import Live

        self._initialized = False
        self.live = None
        if isinstance(live, Live):
            self.live = live
        elif live is not None:
            raise RuntimeError(f"Found class {live.__class__} for live, expected dvclive.Live")

        self._log_model = log_model
        if self._log_model is None:
            log_model_env = os.getenv("HF_DVCLIVE_LOG_MODEL", "FALSE")
            if log_model_env.upper() in ENV_VARS_TRUE_VALUES:
                self._log_model = True
            elif log_model_env.lower() == "all":
                self._log_model = "all"

    def setup(self, args, state, model):
        """
        Setup the optional DVCLive integration. To customize this callback beyond the environment variables below, see
        [here](https://dvc.org/doc/dvclive/ml-frameworks/huggingface).

        Environment:
        - **HF_DVCLIVE_LOG_MODEL** (`str`, *optional*):
            Whether to use `dvclive.Live.log_artifact()` to log checkpoints created by [`Trainer`]. If set to `True` or
            *1*, the final checkpoint is logged at the end of training. If set to `all`, the entire
            [`TrainingArguments`]'s `output_dir` is logged at each checkpoint.
        """
        from dvclive import Live

        self._initialized = True
        if state.is_world_process_zero:
            if not self.live:
                self.live = Live()
            self.live.log_params(args.to_dict())

    def on_train_begin(self, args, state, control, model=None, **kwargs):
        if not self._initialized:
            self.setup(args, state, model)

    def on_log(self, args, state, control, model=None, logs=None, **kwargs):
        if not self._initialized:
            self.setup(args, state, model)
        if state.is_world_process_zero:
            from dvclive.plots import Metric
            from dvclive.utils import standardize_metric_name

            for key, value in logs.items():
                if Metric.could_log(value):
                    self.live.log_metric(standardize_metric_name(key, "dvclive.huggingface"), value)
                else:
                    logger.warning(
                        "Trainer is attempting to log a value of "
                        f'"{value}" of type {type(value)} for key "{key}" as a scalar. '
                        "This invocation of DVCLive's Live.log_metric() "
                        "is incorrect so we dropped this attribute."
                    )
            self.live.next_step()

    def on_save(self, args, state, control, **kwargs):
        if self._log_model == "all" and self._initialized and state.is_world_process_zero:
            self.live.log_artifact(args.output_dir)

    def on_train_end(self, args, state, control, **kwargs):
        if self._initialized and state.is_world_process_zero:
            from transformers.trainer import Trainer

            if self._log_model is True:
                fake_trainer = Trainer(
                    args=args,
                    model=kwargs.get("model"),
                    processing_class=kwargs.get("tokenizer"),
                    eval_dataset=["fake"],
                )
                name = "best" if args.load_best_model_at_end else "last"
                output_dir = os.path.join(args.output_dir, name)
                fake_trainer.save_model(output_dir)
                self.live.log_artifact(output_dir, name=name, type="model", copy=True)


SUMMARY:
**Summary: DVCLiveCallback Class**

The `DVCLiveCallback` class is a[`TrainerCallback`] that sends logs to [DVCLIVE](https://www.dvc.org/doc/dvclive). It integrates with DVCLIVE by logging model checkpoints and metrics.

**Key Features**

* Configurable through environment variables (`HF_DVCLIVE_LOG_MODEL`)
* Logs model checkpoints using `dvclive.Live.log_artifact()`
* Logs training progress metrics using `dvclive.Live.log_metric()`
* Can log entire output directory at each checkpoint (if `log_model = "all"`)

**Methods**

* `__init__()`: Initializes the callback with `Live` instance and configures logging
* `setup(args, state, model)`: Setup optional DVCLIVE integration
* `on_train_begin()`: Sets up DVCLIVE if not initialized
* `on_log()`: Logs training progress metrics and moves to next step in DVCLIVE
* `on_save()``: Logs entire output directory at each checkpoint (if `log_model = "all"`)
* `on_train_end()`: Logs final modelCheckpoint and ends DVCLIVE session

**Dependencies**

* `dvclive.Live`
* `hvclive.Trainer`


key: chroma:document
string_value:
CODE:

def get_reporting_integration_callbacks(report_to):
    for integration in report_to:
        if integration not in INTEGRATION_TO_CALLBACK:
            raise ValueError(
                f"{integration} is not supported, only {', '.join(INTEGRATION_TO_CALLBACK.keys())} are supported."
            )



SUMMARY:
**Searchable and Contextual Summary**

`get_reporting_integration_callbacks(report_to)`

**Purpose**: Returns list of callbacks associated with given reporting integrations.

**Functionality**:

* Iterates through `report_to` and checks if each integration is supported by `INTEGRATION_TO_CALLBACK`.
* Raises ValueError if unsupported integration found.
* Returns a list of corresponding callbacks for supported integrations using dictionary lookup (`INTEGRATION_TO_CALLBACK[integration]`).

**Key Implementation Details**:

* Uses list comprehension to create return list from dictionary values.
* Utilizes `INIntegration_TO_CALLBACK` dictionary to retrieve callbacks.

**Notable Dependencies and Requirements**:
```python
from INTEGRATION_TO_CALLBACK import INTEGRATION_TO_CALLBACK
```
Note: This summary assumes that `INTEGRATION_TO_CALLBACK` is a defined dictionary elsewhere in the codebase. If it's not, additional context would be needed to provide a more comprehensive summary.


key: chroma:document
string_value:
CODE:

def flash_attention_forward(
    module: torch.nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    dropout: float = 0.0,
    scaling: Optional[float] = None,
    sliding_window: Optional[int] = None,
    softcap: Optional[float] = None,
    **kwargs,
) -> Tuple[torch.Tensor, None]:
    # This is before the transpose
    seq_len = query.shape[2]

    # FA2 uses non-transposed inputs
    query = query.transpose(1, 2)
    key = key.transpose(1, 2)
    value = value.transpose(1, 2)

    # In PEFT, usually we cast the layer norms in float32 for training stability reasons
    # therefore the input hidden states gets silently casted in float32. Hence, we need
    # cast them back in the correct dtype just to be sure everything works as expected.
    # This might slowdown training & inference so it is recommended to not cast the LayerNorms
    # in fp32. (usually our RMSNorm modules handle it correctly)
    target_dtype = None
    if query.dtype == torch.float32:
        if torch.is_autocast_enabled():
            target_dtype = torch.get_autocast_gpu_dtype()
        # Handle the case where the model is quantized
        elif hasattr(module.config, "_pre_quantization_dtype"):
            target_dtype = module.config._pre_quantization_dtype
        else:
            target_dtype = next(layer for layer in module.modules() if isinstance(layer, torch.nn.Linear)).weight.dtype

    # FA2 always relies on the value set in the module, so remove it if present in kwargs to avoid passing it twice
    kwargs.pop("is_causal", None)

    attn_output = _flash_attention_forward(
        query,
        key,
        value,
        attention_mask,
        query_length=seq_len,
        is_causal=module.is_causal,
        dropout=dropout,
        softmax_scale=scaling,
        sliding_window=sliding_window,
        softcap=softcap,
        use_top_left_mask=_use_top_left_mask,
        target_dtype=target_dtype,
        **kwargs,
    )



SUMMARY:
**Function Summary: `flash_attention_forward`**

The `flash_attention_forward` function implements the Flash Attention mechanism in PyTorch. It forward-passes attention weights from input query, key, and value tensors to produce output attention weights.

**Primary Purpose**
Implementates Flash Attention module, producing attention weights based on dot product of query and key tensors.

**Key Implementation Details**

* Transposes input tensors (query, key, value) for FA2 compatibility.
* Adjusts data types for training stability and model configuration.
* Removes unnecessary `is_causal` flag from keyword arguments.

**Important Method Signatures and Interfaces**

* `_flash_attention_forward`: an internal function taking query, key, value, attention_mask, and several flags as inputs.

External Dependencies/Tech Requirements:

* PyTorch (`torch`)
* Torch Autocasting (`torch.is_autocast_enabled`, `torch.get-autocast-gpu-dtype`)

Note: The summary has preserved the class/function/variable names exactly, as per guidelines.


key: chroma:document
string_value:
CODE:

class FbgemmFp8Linear(torch.nn.Module):
    def __init__(self, in_features, out_features, bias, weight_dtype=torch.float32):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features

        self.register_buffer("weight", torch.zeros((out_features, in_features), dtype=torch.float8_e4m3fn))
        self.register_buffer("weight_scale", torch.zeros((out_features, 1), dtype=weight_dtype))
        self.register_buffer("input_scale_ub", torch.zeros([1], dtype=torch.float), persistent=False)

        if bias:
            self.register_buffer("bias", torch.zeros((self.out_features), dtype=weight_dtype))
        else:
            self.bias = None

    def forward(self, x):
        num_tokens = None
        # quantize_fp8_per_row will squash the leading dimensions, so save the desired shape here
        output_shape = (*x.shape[:-1], -1)
        # x_quantized and x_scale are not necessarily on the same device as x, this is an issue.
        # https://github.com/pytorch/FBGEMM/blob/e08af8539c391437f447173863df0f3f6f6f1855/fbgemm_gpu/experimental/gen_ai/src/quantize/quantize.cu#L1237C3-L1237C45
        x_quantized, x_scale = torch.ops.fbgemm.quantize_fp8_per_row(
            x.view(-1, x.shape[-1]), num_tokens, self.input_scale_ub
        )
        # moving x_quantized, x_scale here creates glibberish output ... However, if we move the output, it works
        # x_quantized, x_scale = x_quantized.to(x.device), x_scale.to(x.device)

        # The computation still happens on the device where self.weight is even if x_quantized is not on the same device as self.weight
        output = torch.ops.fbgemm.f8f8bf16_rowwise(
            x_quantized, self.weight, x_scale, self.weight_scale, use_fast_accum=True
        )
        output = output + self.bias if self.bias is not None else output
        # Hacky for now, we have the output to the device of x
        output = output.to(x.device)
        output = output.reshape(output_shape)
        del x_quantized, x_scale


SUMMARY:
**FbgemmFp8Linear Class Summary**

### Main Purpose and Functionality

The `FbgemmFp8Linear` class is a PyTorch implementation of the FBGEMM (Flexible Binary GEMM) algorithm for linear transformations. It specializes in quantized floating-point operations using half precision ( fp8).

**Key Implementation Details:**

* The forward pass performs row-wise multiplication and addition:
	+ `x_quantized`, `x_scale` are obtained by calling `torch.ops.fbgemm.quantize_fp8_per_row`
	+ The result is then processed in the device where `self.weight` resides
* The computation involves two main operations: 
  - `torch.ops.fbgemm.f8f8bf16_rowwise(x_quantized, self.weight, x_scale, self.weight_scale)`: performs row-wise multiplication of `xQuantized` and `self.weight`
  - Adding bias Term (`output + self.bias`) if available

**Notable Dependencies:**

* `torch.ops.fbgemm.quantize_fp8_per_row` and `torch.ops.fbgemm.f8f8bf16_rowwise`: FBGEMM implementation providing optimized quantized multiplication
* `torch.float8_e4m3fn` (dtype): half precision data type used for weight storage

**Additional Details:**

* The class inherits from PyTorch's `nn.Module`
* It requires the input tensor to have shape compatible with row-wise broadcasting, i.e., `(2, x, ...) -> ...`


key: chroma:document
string_value:
CODE:

def _replace_with_fbgemm_fp8_linear(
    model,
    modules_to_not_convert=None,
    current_key_name=None,
    quantization_config=None,
    has_been_replaced=False,
    pre_quantized=False,
):
    """
    Private method that wraps the recursion for module replacement.

    Returns the converted model and a boolean that indicates if the conversion has been successfull or not.
    """
    if current_key_name is None:
        current_key_name = []

    for name, module in model.named_children():
        current_key_name.append(name)

        if (isinstance(module, nn.Linear)) and name not in modules_to_not_convert:
            # Check if the current key is not in the `modules_to_not_convert`
            current_key_name_str = ".".join(current_key_name)
            if not any(
                (key + "." in current_key_name_str) or (key == current_key_name_str) for key in modules_to_not_convert
            ):
                with init_empty_weights(include_buffers=True):
                    in_features = module.in_features
                    out_features = module.out_features
                    model._modules[name] = FbgemmFp8Linear(
                        in_features,
                        out_features,
                        module.bias is not None,
                    )
                    has_been_replaced = True

                    # Force requires grad to False to avoid unexpected errors
                    model._modules[name].requires_grad_(False)
                # set non persistant buffer outside of init_empty_weights
                model._modules[name].input_scale_ub = torch.tensor(
                    [quantization_config.activation_scale_ub], dtype=torch.float
                )
        if len(list(module.children())) > 0:
            _, has_been_replaced = _replace_with_fbgemm_fp8_linear(
                module,
                modules_to_not_convert,
                current_key_name,
                quantization_config,
                has_been_replaced=has_been_replaced,
                pre_quantized=pre_quantized,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Summary:**

This private method, `_replace_with_fbgemm_fp8_linear`, wraps recursive replacements of linear layers in a PyTorch model with FbgemmFP8Linear modules. It:

* Iterates through the model's children and checks if each Linear module should be replaced.
* If not in `modules_to_not_convert`, it creates an FbgemmFP8Linear wrapper around the original module, setting its input_scale_ub and requires_grad properties.

**Key Details:**

* Replaces PyTorch Linear modules with FbgemmFP8Linear, preserving bias information.
* Recursively replaces child modules if they are also Linear.
* Use of `modules_to_not_convert` to specify excluded modules, and `quantization_config` to set input_scale_ub.

**Notable Interfaces/Dependencies:**

* PyTorch's Neural Networks (nn) module for accessing Linear layers.
* FbgemmFP8Linear wrappers from FBOSS framework for replacing Linear modules.


key: chroma:document
string_value:
CODE:

def replace_with_fbgemm_fp8_linear(
    model, modules_to_not_convert=None, current_key_name=None, quantization_config=None, pre_quantized=False
):
    """
    A helper function to replace all `torch.nn.Linear` modules by `FbgemmFp8Linear` modules.
    This will enable running your models using high performance fp8 kernel from FBGEMM library.

    The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should
    be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no
    CPU/GPU memory is required to run this function. Each weight will be quantized along the channel.

    Parameters:
        model (`torch.nn.Module`):
            Input model or `torch.nn.Module` as the function is run recursively.
        modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`):
            Names of the modules to not convert in `FP8Linear`. In practice we keep the `lm_head` in full precision
            for numerical stability reasons.
        current_key_name (`List[`str`]`, *optional*):
            An array to track the current key of the recursion. This is used to check whether the current key (part of
            it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or
            `disk`).
    """

    modules_to_not_convert = ["lm_head"] if modules_to_not_convert is None else modules_to_not_convert

    if quantization_config.modules_to_not_convert is not None:
        modules_to_not_convert.extend(quantization_config.modules_to_not_convert)
    modules_to_not_convert = list(set(modules_to_not_convert))
    model, has_been_replaced = _replace_with_fbgemm_fp8_linear(
        model, modules_to_not_convert, current_key_name, quantization_config, pre_quantized=pre_quantized
    )

    if not has_been_replaced:
        logger.warning(
            "You are loading your model using FP8 quantization but no linear modules were found in your model."
            " Please double check your model architecture, or submit an issue on github if you think this is"
            " a bug."
        )



SUMMARY:
**Summary:**

Represents a function to replace `torch.nn.Linear` modules with `FbgemmFp8Linear` modules in a given model, enabling fp8 quantization and high-performance kernel usage.

**Key Implementation Details:**

* Replaces all `torch.nn.Linear` modules except for `lm_head`, recursively traversing the model's structure.
* Applies `init_empty_weights` context manager for memory efficiency.
* Quantizes each weight along the channel axis.
* Extends `modules_to_not_convert` list with additional module names from `quantization_config.modules_to_not_convert`.
* Checks and logs warnings if no linear modules are found in the model.

**Method Signatures and Interfaces:**

* `_replace_with_fbgemm_fp8_linear`: Recursive wrapper function to perform the replacement.
* `model` (input): The input model or torch.nn.Module instance.
* `quantization_config`: Optional object containing configuration for quantization.

**Notable Dependencies and Requirements:**

* `torch.nn.Linear`
* `FbgemmFp8Linear` module
* `init_empty_weights` context manager
* `logger` (logging mechanism)
* `quantization_config` (optional)


key: chroma:document
string_value:
CODE:

def _replace_with_eetq_linear(
    model,
    modules_to_not_convert=None,
    current_key_name=None,
    quantization_config=None,
    has_been_replaced=False,
    pre_quantized=False,
):
    """
    Private method that wraps the recursion for module replacement.

    Returns the converted model and a boolean that indicates if the conversion has been successfull or not.
    """
    if current_key_name is None:
        current_key_name = []

    for name, module in model.named_children():
        current_key_name.append(name)

        if (isinstance(module, nn.Linear)) and name not in modules_to_not_convert:
            # Check if the current key is not in the `modules_to_not_convert`
            current_key_name_str = ".".join(current_key_name)
            if not any(
                (key + "." in current_key_name_str) or (key == current_key_name_str) for key in modules_to_not_convert
            ):
                with init_empty_weights():
                    in_features = module.in_features
                    out_features = module.out_features
                    model._modules[name] = eetq.EetqLinear(
                        in_features, out_features, module.bias is not None, module.weight.device
                    )
                    if pre_quantized:
                        model._modules[name].register_scale(module.weight.device)
                    has_been_replaced = True

                    # Force requires grad to False to avoid unexpected errors
                    model._modules[name].requires_grad_(False)
        if len(list(module.children())) > 0:
            _, has_been_replaced = _replace_with_eetq_linear(
                module,
                modules_to_not_convert,
                current_key_name,
                quantization_config,
                has_been_replaced=has_been_replaced,
                pre_quantized=pre_quantized,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Function Summary: `_replace_with_eetq_linear`**

**Purpose:** This private method recursively replaces PyTorch linear layers with EETQLinear modules in a given model architecture.

**Key Implementation Details:**

* Iterates through the model's children (modules) and replaces Linear layers not in `modules_to_not_convert` list.
* Customizes EETQLinear initialization with:
	+ In-plane features
	+ Out-plane features
	+ Bias existence flag
	+ Weight device
	+ Pre-quantization activation (if enabled)
* Recursively calls `_replace_with_eetq_linear` on child modules with updated `has_been_replaced` flag.
* Returns converted model and success flag.

**Notable Dependencies/Requirements:**

* PyTorch (`nn.Linear`, `model.named_children`)
* EETQLinear module (custom implementation)

**Method Signatures and Interfaces:**

* `eetq.EetqLinear(in_features, out_features, bias_is_not_none, weight_device)`
* `init_empty_weights()`

Please note that the summary is concise while preserving key details. If there's a specific requirement for further analysis or additional information, let me know!


key: chroma:document
string_value:
CODE:

def replace_with_eetq_linear(
    model, modules_to_not_convert=None, current_key_name=None, quantization_config=None, pre_quantized=False
):
    """
    A helper function to replace all `torch.nn.Linear` modules by `eetq.EetqLinear` modules from the `eetq`
    library. This will enable running your models using high performance int8 weight-only gemm kerner from
    FasterTransformer and TensorRT-LLM. Make sure `eetq` compiled with the correct CUDA
    version of your hardware is installed before running this function. EETQ shall be installed via the source
    'https://github.com/NetEase-FuXi/EETQ'

    The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should
    be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no
    CPU/GPU memory is required to run this function. Each weight will be quantized along the channel.

    Parameters:
        model (`torch.nn.Module`):
            Input model or `torch.nn.Module` as the function is run recursively.
        modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`):
            Names of the modules to not convert in `EetqLinear`. In practice we keep the `lm_head` in full precision
            for numerical stability reasons.
        current_key_name (`List[`str`]`, *optional*):
            An array to track the current key of the recursion. This is used to check whether the current key (part of
            it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or
            `disk`).
    """

    modules_to_not_convert = ["lm_head"] if modules_to_not_convert is None else modules_to_not_convert

    if quantization_config.modules_to_not_convert is not None:
        modules_to_not_convert.extend(quantization_config.modules_to_not_convert)
    modules_to_not_convert = list(set(modules_to_not_convert))
    model, has_been_replaced = _replace_with_eetq_linear(
        model, modules_to_not_convert, current_key_name, quantization_config, pre_quantized=pre_quantized
    )

    if not has_been_replaced:
        logger.warning(
            "You are loading your model using eetq but no linear modules were found in your model."
            " Please double check your model architecture, or submit an issue on github if you think this is"
            " a bug."
        )



SUMMARY:
**Function Summary: `replace_with_eetq_linear`**

Replaces all `torch.nn.Linear` modules in the given `model` with `eetq.EetqLinear` modules, enabling high-performance int8 weight-only gemm kernel usage.

**Key Implementation Details:**

* Replaces specified patterns of `torch.nn.Linear` modules except for `lm_head`, which remains unchanged.
* Uses recursion to replace modules in the model tree.
* Quantizes weights along channels within replaced modules.
* Utilizes the `_replace_with_eetq_linear` recursive helper function.

**Notable Dependencies/Requirements:**

* `eetq.EetqLinear` module from the EETQ library (`https://github.com/NetEase-FuXi/EETQ`)
* Correct CUDA version installed on hardware.

**Method Signatures/Interfaces:**

* `_replace_with_eetq_linear(model, modules_to_not_convert, current_key_name, quantization_config, pre_quantized)`: Recursive helper function for replacing `torch.nn.Linear` modules.


key: chroma:document
string_value:
CODE:

def pack_weights(quantized_weights: torch.Tensor) -> torch.Tensor:
    """
    Packs a tensor of quantized weights into a compact format using 2 bits per value.

    Parameters:
    -----------
    quantized_weights : torch.Tensor
        A tensor containing ternary quantized weights with values in {-1, 0, 1}. These values are adjusted to
        {0, 1, 2} before being packed.

    Returns:
    --------
    torch.Tensor
        A packed tensor where each element stores 4 quantized values (each using 2 bits) in an 8-bit format.
    """

    original_shape = quantized_weights.shape

    row_dim = (original_shape[0] + VALUES_PER_ITEM - 1) // VALUES_PER_ITEM

    if len(original_shape) == 1:
        packed_tensor_shape = (row_dim,)
    else:
        packed_tensor_shape = (row_dim, *original_shape[1:])

    quantized_weights += 1
    packed = torch.zeros(packed_tensor_shape, device=quantized_weights.device, dtype=torch.uint8)
    unpacked = quantized_weights.to(torch.uint8)

    it = min(VALUES_PER_ITEM, (original_shape[0] // row_dim) + 1)
    for i in range(it):
        start = i * row_dim
        end = min(start + row_dim, original_shape[0])
        packed[: (end - start)] |= unpacked[start:end] << 2 * i



SUMMARY:
**Summary**

`pack_weights` function packs a tensor of quantized weights into an compact 8-bit format using 2 bits per value.

**Key Features:**

* Inputs: `quantized_weights` (torch.Tensor) - Ternary quantized weights in {-1, 0, 1}
* Outputs: `packed_tensor` (torch.Tensor) - Packed tensor storing 4 values (each using 2 bits) in an 8-bit format
* Function:
	+ Packs ternary to quaternary mapping: {(-1), 0, 1} -> {0, 1, 2}
	+ Uses bit-packing to store multiple values (4) in a single byte (using 2 bits per value)
	+ Assumes input tensor has shape `(num_rows, num_cols)` or `(num_rows,)`

**Implementation Details:**

* Packed using bitwise OR operation
* Bit shift used to insert packed values into corresponding bytes of the output tensor


key: chroma:document
string_value:
CODE:

class BitLinear(nn.Module):
    def __init__(self, in_features: int, out_features: int, bias: bool, device=None, dtype=None):
        super().__init__()
        self.dtype = dtype
        self.in_features = in_features
        self.out_features = out_features
        self.register_buffer(
            "weight",
            torch.zeros(
                (out_features // VALUES_PER_ITEM, in_features),
                dtype=torch.uint8,
                device=device,
            ),
        )
        self.register_buffer(
            "weight_scale",
            torch.ones(
                (1),
                dtype=dtype,
                device=device,
            ),
        )
        if bias:
            self.register_buffer("bias", torch.zeros((out_features), dtype=dtype, device=device))
        else:
            self.bias = None

    @torch.compile
    def activation_quant(self, input, num_bits=8):
        """
        Activation function : Performs symmetric, per-token quantization on the input activations.
        Parameters:
        -----------
        x : torch.Tensor
            Input activations to be quantized.
        num_bits : int, optional (default=8)
            Number of bits to use for quantization, determining the quantization range.

        Returns:
        --------
        result : torch.Tensor
            Quantized activation tensor, with values mapped to an `int8` range.
        scale : torch.Tensor
            The per-channel scaling factors used to quantize the tensor.
        """
        Qn = -(2 ** (num_bits - 1))
        Qp = 2 ** (num_bits - 1) - 1
        scale = Qp / input.abs().max(dim=-1, keepdim=True).values.clamp(min=1e-5)
        result = (input * scale).round().clamp(Qn, Qp)
        return result.to(torch.int8), scale

    @torch.compile
    def post_quant_process(self, input, input_scale, weight_scale):
        out = input / (input_scale * weight_scale)
        return out

    def forward(self, input):
        w = self.weight
        w_quant = unpack_weights(w, dtype=self.dtype)
        input_quant, input_scale = self.activation_quant(input)
        y = F.linear(input_quant.to(self.dtype), w_quant)
        y = self.post_quant_process(y, self.weight_scale, input_scale)
        if self.bias is not None:
            y += self.bias.view(1, -1).expand_as(y)


SUMMARY:
**BitLinear Neural Network Module**

**Purpose:**
A PyTorch neural network module for efficient and quantized linear activation functions.

**Key Implementation Details:**

*   **Weight Quantization:** The `activation_quant` method performs symmetric, per-token quantization on the input activations using a specified number of bits.
*   **Post-Quantization Processing:** The `post_quant_process` method normalizes the output by scaling down the linear activation with the inverse of the sum of the weight scaling factor and the input scaling factor.
*   **Weight Buffer Registration:** The module registers two buffers for the quantized weights and bias (if present).
*   **Neural Forward Pass:**

    *   Extracts the quantized weights using `unpack_weights`.
    *   Applies the quantization activation function to the input.
    *   Performs linear transformation using the quantized weights.
    *   Post-processes the output by normalizing it with the inverse of the sum of the weight scaling factor and the input scaling factor.

**Dependences:**

*   PyTorch Tensor Operations
*   F.linear function
*   unpack_weights function

```python
class BitLinear(nn.Module):
    def __init__(self, in_features: int, out_features: int, bias: bool, device=None, dtype=None) -> None:
        super().__init__()
        self.dtype = dtype
        self.in_features = in_features
        self.out_features = out_features

        # Register buffers for weights and (optional) bias
        self.register_buffer("weight", torch.zeros((out_features // VALUES_PER_ITEM, in_features), dtype=torch.uint8, device=device))
        self.register_buffer("weight_scale", torch.ones((1), dtype=dtype, device=device))

        if bias:
            self.register_buffer("bias", torch.zeros((out_features), dtype=dtype, device=device))
        else:
            self.bias = None

    def activation_quant(self, input: torch.Tensor, num_bits: int = 8) -> (torch.Tensor, torch.Tensor):
        # ... (same implementation)

    def post_quant_process(self, input: torch.Tensor, input_scale: torch.Tensor, weight_scale: torch.Tensor) -> torch.Tensor:
        # ... (same implementation)

    def forward(self, input: torch.Tensor) -> torch.Tensor:
        w = self.weight
        w_quant = unpack_weights(w, dtype=self.dtype)
        input_quant, input_scale = self.activation_quant(input)
        y = F.linear(input_quant.to(self.dtype), w_quant)
        y = self.post_quant_process(y, self.weight_scale, input_scale)

        # Apply bias if present
        return (
            y + self.bias.view(1, -1).expand_as(y) if self.bias is not None else y
        )
```

Note: The `unpack_weights` function is assumed to be defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def _replace_with_bitnet_linear(
    model,
    modules_to_not_convert=None,
    current_key_name=None,
    quantization_config=None,
    has_been_replaced=False,
    pre_quantized=False,
):
    """
    Private method that wraps the recursion for module replacement.

    Returns the converted model and a boolean that indicates if the conversion has been successfull or not.
    """

    if current_key_name is None:
        current_key_name = []

    for name, module in model.named_children():
        if current_key_name is None:
            current_key_name = []
        current_key_name.append(name)

        # Check if the current key is not in the `modules_to_not_convert`
        if not any(key in ".".join(current_key_name) for key in modules_to_not_convert):
            with init_empty_weights():
                if isinstance(module, nn.Linear) and name not in modules_to_not_convert:
                    in_features = module.in_features
                    out_features = module.out_features
                    model._modules[name] = BitLinear(
                        in_features=in_features,
                        out_features=out_features,
                        bias=module.bias is not None,
                        device=module.weight.device,
                        dtype=module.weight.dtype,
                    )
                    has_been_replaced = True
                    model._modules[name].requires_grad_(False)

        if len(list(module.children())) > 0:
            _, has_been_replaced = _replace_with_bitnet_linear(
                module,
                modules_to_not_convert=modules_to_not_convert,
                current_key_name=current_key_name,
                quantization_config=quantization_config,
                has_been_replaced=has_been_replaced,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Function Summary: `_replace_with_bitnet_linear`**

* **Purpose**: Recursively replaces PyTorch linear layers with BitNetLinear equivalents while preserving the original model structure.
* **Key Implementation Details**:
	+ Uses a recursive approach to traverse the model's child modules.
	+ Checks if each module is replaceable before performing the replacement.
	+ Creates new instances of BitLinear, initializing weights and bias as necessary.
* **Dependencies**:
	+ `model`: The input PyTorch model.
	+ `nn.Linear` and `bitnet.linear.BitLinear`
* **Interfaces Implemented**: None.

This function is part of a larger codebase aimed at converting models from other frameworks to BitNet, which provides efficient neural network computing.


key: chroma:document
string_value:
CODE:

def replace_with_bitnet_linear(
    model,
    modules_to_not_convert=None,
    current_key_name=None,
    quantization_config=None,
    pre_quantized=False,
):
    """
    A helper function to replace all `torch.nn.Linear` modules by `BitLinear158` modules`.

    The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should
    be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no
    CPU/GPU memory is required to run this function. Each weight will be quantized along the channel.

    Parameters:
        model (`torch.nn.Module`):
            Input model or `torch.nn.Module` as the function is run recursively.
        modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`):
            Names of the modules to not convert in `EetqLinear`. In practice we keep the `lm_head` in full precision
            for numerical stability reasons.
        current_key_name (`List[`str`]`, *optional*):
            An array to track the current key of the recursion. This is used to check whether the current key (part of
            it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or
            `disk`).
    """
    modules_to_not_convert = ["lm_head"] if modules_to_not_convert is None else modules_to_not_convert
    if quantization_config and quantization_config.modules_to_not_convert is not None:
        modules_to_not_convert.extend(quantization_config.modules_to_not_convert)
    modules_to_not_convert = list(set(modules_to_not_convert))
    model, has_been_replaced = _replace_with_bitnet_linear(
        model,
        modules_to_not_convert,
        current_key_name,
        quantization_config,
        pre_quantized=pre_quantized,
    )

    if not has_been_replaced:
        logger.warning(
            "You are loading your model using bitnet but no linear modules were found in your model."
            " Please double check your model architecture, or submit an issue on github if you think this is"
            " a bug."
        )



SUMMARY:
**Summary**

`replace_with_bitnet_linear` function: Recursively replaces `torch.nn.Linear` modules with `BitLinear158` in a given model, preserving specific modules. It quantizes weights along channels.

**Key Details**

* Primary purpose: Replace `torch.nn Linear` with `BitLinear158`.
* Key algorithm: Recursive replacement under `init_empty_weights` context manager.
* Notable dependencies:
	+ `torch.nn.Linear`
	+ `BitLinear158`
	+ `quantization_config` with optional `modules_to_not_convert` attribute.

**Implementation Focus**

* Replaces specific modules (`lm_head`, specified by `modules_to_not_convert`) while preserving others.
* Quantizes weights along channels using `_quantization_config`.


key: chroma:document
string_value:
CODE:

def replace_with_aqlm_linear(
    model,
    quantization_config=None,
    linear_weights_not_to_quantize=None,
    current_key_name=None,
    has_been_replaced=False,
):
    """
    Public method that recursively replaces the Linear layers of the given model with AQLM quantized layers.
    `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the
    conversion has been successfull or not.

    Args:
        model (`torch.nn.Module`):
            The model to convert, can be any `torch.nn.Module` instance.
        quantization_config (`AqlmConfig`):
            The quantization config object that contains the quantization parameters.
        linear_weights_not_to_quantize (`list[str]`, *optional*):
            A list of nn.Linear weights to not convert. If a parameter path is in the list (e.g. `lm_head.weight`), the corresponding module will not be
            converted.
        current_key_name (`list`, *optional*):
            A list that contains the current key name. This is used for recursion and should not be passed by the user.
        has_been_replaced (`bool`, *optional*):
            A boolean that indicates if the conversion has been successful or not. This is used for recursion and
            should not be passed by the user.
    """
    if not is_aqlm_available():
        raise ValueError("AQLM is not available. Please install it with `pip install aqlm[cpu,gpu]`")

    if not is_accelerate_available():
        raise ValueError(
            f"AQLM requires Accelerate to be installed: `pip install 'accelerate>={ACCELERATE_MIN_VERSION}'`"
        )

    if linear_weights_not_to_quantize is None:
        linear_weights_not_to_quantize = []

    from accelerate import init_empty_weights
    from aqlm import QuantizedLinear

    for name, module in model.named_children():
        if current_key_name is None:
            current_key_name = []
        current_key_name.append(name)

        if isinstance(module, nn.Linear):
            # Check if the current key is not in the `linear_weights_not_to_quantize`
            if ".".join(current_key_name) + ".weight" not in linear_weights_not_to_quantize:
                with init_empty_weights():
                    in_features = module.in_features
                    out_features = module.out_features

                    model._modules[name] = QuantizedLinear(
                        in_features,
                        out_features,
                        bias=module.bias is not None,
                        in_group_size=quantization_config.in_group_size,
                        out_group_size=quantization_config.out_group_size,
                        num_codebooks=quantization_config.num_codebooks,
                        nbits_per_codebook=quantization_config.nbits_per_codebook,
                    )
                    has_been_replaced = True

                    # Store the module class in case we need to transpose the weight later
                    model._modules[name].source_cls = type(module)
                    # Force requires grad to False to avoid unexpected errors
                    model._modules[name].requires_grad_(False)
        if len(list(module.children())) > 0:
            _, has_been_replaced = replace_with_aqlm_linear(
                module,
                quantization_config=quantization_config,
                linear_weights_not_to_quantize=linear_weights_not_to_quantize,
                current_key_name=current_key_name,
                has_been_replaced=has_been_replaced,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Public Method:** `replace_with_aqlm_linear`
```python
# Recursively replaces Linear layers in a PyTorch model with AQLM quantized layers.

def replace_with_aqlm_linear(
    model: torch.nn.Module,
    quantization_config: AqlmConfig = None,  # Optional Quantization config object
    linear_weights_not_to_quantize: list[str] = None,  # List of nn.Linear weights to not convert
    current_key_name: list = None,  # Current key name for recursion (user-specified)
    has_been_replaced: bool = False,  # Flag indicating successful conversion (user-specified)
) -> tuple:
```

**Key Features:**

*   **Quantized Linear Layers:** Replaces `torch.nn.Linear` layers with AQLM quantized layers.
*   **Recursive Conversion:** Uses recursion to convert all Linear layers in the given model.

**Functionality:**

1.  Checks availability of necessary libraries (AQLM and Accelerate).
2.  Iterates through the model's children, identifying `nn.Linear` modules eligible for conversion.
3.  Initializes AQLM quantized linear layer with specified configuration and weights if not excluded by `linear_weights_not_to_quantize`.
4.  Recursively calls itself for child modules (if any).

**Implementation Details:**

*   Uses `aqlm` QuantizedLinear class to create quantized layers.
*   `accelerate` is used to initialize empty weights for the conversion process.
*   Maintains a flag (`has_been_replaced`) indicating the success of each recursive call.

Returns a tuple containing the converted model and the updated `has_been_replaced` flag.


key: chroma:document
string_value:
CODE:

def replace_with_vptq_linear(
    model,
    quantization_config=None,
    modules_to_not_convert=None,
    current_key_name=None,
    has_been_replaced=False,
):
    """
    Public method that recursively replaces the Linear layers of the given model with VPTQ quantized layers.
    `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the
    conversion has been successfull or not.

    Args:
        model (`torch.nn.Module`):
            The model to convert, can be any `torch.nn.Module` instance.
        quantization_config (`VptqConfig`):
            The quantization config object that contains the quantization parameters.
        modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`):
            Names of the modules to not convert in `VQuantLinear`. In practice we keep the `lm_head` in full precision
            for numerical stability reasons.
        current_key_name (`list`, *optional*):
            A list that contains the current key name. This is used for recursion and should not be passed by the user.
        has_been_replaced (`bool`, *optional*):
            A boolean that indicates if the conversion has been successful or not. This is used for recursion and
            should not be passed by the user.
    """

    modules_to_not_convert = ["lm_head"] if not modules_to_not_convert else modules_to_not_convert

    for name, module in model.named_children():
        if current_key_name is None:
            current_key_name = []
        current_key_name.append(name)
        layer_name = ".".join(current_key_name)
        shared_layer_config = quantization_config.shared_layer_config
        config_for_layers = quantization_config.config_for_layers

        if (
            isinstance(module, nn.Linear)
            and layer_name not in modules_to_not_convert
            and ((layer_name in config_for_layers) or (current_key_name[-1] in shared_layer_config))
        ):
            layer_params = config_for_layers.get(layer_name, None) or shared_layer_config.get(
                current_key_name[-1], None
            )

            with init_empty_weights():
                in_features = module.in_features
                out_features = module.out_features

                model._modules[name] = VQuantLinear(
                    in_features,
                    out_features,
                    vector_lens=layer_params["vector_lens"],
                    num_centroids=layer_params["num_centroids"],
                    num_res_centroids=layer_params["num_res_centroids"],
                    group_num=layer_params["group_num"],
                    group_size=layer_params["group_size"],
                    outlier_size=layer_params["outlier_size"],
                    indices_as_float=layer_params["indices_as_float"],
                    enable_norm=layer_params["enable_norm"],
                    enable_perm=layer_params["enable_perm"],
                    is_indice_packed=True,
                    enable_proxy_error=False,
                    bias=module.bias is not None,
                )
                has_been_replaced = True

                # Force requires grad to False to avoid unexpected errors
                model._modules[name].requires_grad_(False)
        if len(list(module.children())) > 0:
            _, has_been_replaced = replace_with_vptq_linear(
                module,
                quantization_config=quantization_config,
                modules_to_not_convert=modules_to_not_convert,
                current_key_name=current_key_name,
                has_been_replaced=has_been_replaced,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Summary:**
The `replace_with_vptq_linear` function recursively replaces Linear layers in a PyTorch model with VPTQ quantized layers from a given configuration. The conversion is done while preserving the numerical stability of certain modules (`lm_head`) and keeping track of success.

**Key Implementation Details:**

* Handles recursive replacement of Linear layers
* Checks for presence in `config_for_layers` or shared_layer_config dictionaries
* Utilizes VQuantLinear module with specific layer parameters
* Preserves `requires_grad` setting to minimize errors

**Notable Dependencies/Requirements:**

* PyTorch (`torch.nn.Module`, `nn.Linear`)
* `VptqConfig` object (referenced by exact name)
* External dependency (`accelerate` is needed, but not explicitly referenced in the given code chunk)

**Primary Purpose:**
Replace Linear layers with VPTQ quantized equivalent in a PyTorch model while maintaining numerical stability.


key: chroma:document
string_value:
CODE:

def is_fsdp_managed_module(module: nn.Module) -> bool:
    if not is_torch_available():
        return False

    import torch.distributed.fsdp

    return isinstance(module, torch.distributed.fsdp.FullyShardedDataParallel) or getattr(
        module, "_is_fsdp_managed_module", False


SUMMARY:
**Summary:**

`is_fsdp_managed_module` function checks if a given PyTorch module is managed by the Fully Sharded Data Parallel (FSDP) algorithm. It returns `True` if the module is an instance of `torch.distributed.fsdp.FullyShardedDataParallel` or has a custom `_is FSFP Managed Module` attribute set.

**Key Details:**

* Primary purpose: Verify FSDP module
* Key implementation details:
	+ Checks for `torch.distributed.fsdp.FullyShardedDataParallel` module type
	+ Checks for custom `_is_fsdp_managed_module` attribute on the input module
* Notable dependencies or requirements: 
    - Torch distributed package (`torch.distributed.fsdp`)
    - PyTorch framework (implied, but not explicitly mentioned)


key: chroma:document
string_value:
CODE:

def replace_with_quanto_layers(
    model,
    quantization_config=None,
    modules_to_not_convert=None,
    current_key_name=None,
    has_been_replaced=False,
):
    """
    Public method that recursively replaces the Linear layers of the given model with Quanto quantized layers.
    Returns the converted model and a boolean that indicates if the conversion has been successfull or not.

    Args:
        model (`torch.nn.Module`):
            The model to convert, can be any `torch.nn.Module` instance.
        quantization_config (`AqlmConfig`, defaults to `None`):
            The quantization config object that contains the quantization parameters.
        modules_to_not_convert (`list`, *optional*, defaults to `None`):
            A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be
            converted.
        current_key_name (`list`, *optional*, defaults to `None`):
            A list that contains the current key name. This is used for recursion and should not be passed by the user.
        has_been_replaced (`bool`, *optional*, defaults to `None`):
            A boolean that indicates if the conversion has been successful or not. This is used for recursion and
            should not be passed by the user.
    """
    from accelerate import init_empty_weights

    if is_optimum_quanto_available():
        from optimum.quanto import QLayerNorm, QLinear, qfloat8, qint2, qint4, qint8

    w_mapping = {"float8": qfloat8, "int8": qint8, "int4": qint4, "int2": qint2}
    a_mapping = {None: None, "float8": qfloat8, "int8": qint8}

    if modules_to_not_convert is None:
        modules_to_not_convert = []

    for name, module in model.named_children():
        if current_key_name is None:
            current_key_name = []
        current_key_name.append(name)

        if not any(key in ".".join(current_key_name) for key in modules_to_not_convert):
            with init_empty_weights():
                if isinstance(module, torch.nn.Linear):
                    model._modules[name] = QLinear(
                        in_features=module.in_features,
                        out_features=module.out_features,
                        bias=module.bias is not None,
                        dtype=module.weight.dtype,
                        weights=w_mapping[quantization_config.weights],
                        activations=a_mapping[quantization_config.activations],
                    )
                    model._modules[name].requires_grad_(False)
                    has_been_replaced = True
                elif isinstance(module, torch.nn.LayerNorm):
                    if quantization_config.activations is not None:
                        model._modules[name] = QLayerNorm(
                            module.normalized_shape,
                            module.eps,
                            module.elementwise_affine,
                            module.bias is not None,
                            activations=a_mapping[quantization_config.activations],
                        )
                        has_been_replaced = True
        if len(list(module.children())) > 0:
            _, has_been_replaced = replace_with_quanto_layers(
                module,
                quantization_config=quantization_config,
                modules_to_not_convert=modules_to_not_convert,
                current_key_name=current_key_name,
                has_been_replaced=has_been_replaced,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Summary:**

Public `replace_with_quanto_layers` method that recursively replaces Linear layers of a given PyTorch model with Quanto quantized layers. Takes into account quantization configuration and specific modules to not convert.

```python
def replace_with_quanto_layers(model, 
                               quantization_config=None, 
                               modules_to_not_convert=None,
                               current_key_name=None,
                               has_been_replaced=False,
                              ):
    # Replace linear layer with QLinear (with specific weights/dtypes/activations)
    # Recursively process submodule children
    pass

# Return converted model and flag indicating success/failure
```

**Key Implementation Details:**

1.  **Recursive Processing**: Utilizes the `replace_with_quanto_layers` method on each submodule child, with recursive calls and updated flags.
2.  **Weight/Dtype/Activation Mapping**: Employs the `w_mapping` and `a_mapping` dictionaries to resolve quantization parameters for QLinear/QLayerNorm construction.

**Notable Dependencies:**

*   `torch`
*   `accelerate.init_empty_weights`
*   `optimum.quanto.QLayerNorm`, `QLinear`, `qfloat8`, `qint2`, `qint4`, `qint8`


key: chroma:document
string_value:
CODE:

def is_deepspeed_available():
    package_exists = importlib.util.find_spec("deepspeed") is not None

    # Check we're not importing a "deepspeed" directory somewhere but the actual library by trying to grab the version
    # AND checking it has an author field in the metadata that is HuggingFace.
    if package_exists:
        try:
            if is_torch_mlu_available():
                _ = importlib_metadata.metadata("deepspeed-mlu")
                return True
            _ = importlib_metadata.metadata("deepspeed")
            return True
        except importlib_metadata.PackageNotFoundError:


SUMMARY:
**Summary**

This function checks if the `deepspeed` library is available and can be used with HuggingFace's `torch_mlu` support. It performs the following checks:

1. Checks if the `deep-speed` package exists using `importlib`.
2. Verifies that it's not importing a directory by attempting to grab metadata from `deepspeed-mlu` and `deepspeed` with an author field matching HuggingFace.

**Key Implementation Details**

* Uses `importlib.util.find_spec()` to check package existence.
* Handles `PackageNotFoundError` when checking for dependency version metadata.
* Checks for `torch_mlu` support using additional import statement (`_ = importlib_metadata.metadata("deepspeed-mlu")`).

**Notable Dependencies/Requirements**

* `deeph-speed` (library)
* `huggingface` (metadata author check)
* `torch-mlu` (dependency, checked via internal function `_is_torch_mlu_available()`)


key: chroma:document
string_value:
CODE:

class HfDeepSpeedConfig(DeepSpeedConfig):
    """
    This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage.

    A `weakref` of this object is stored in the module's globals to be able to access the config from areas where
    things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore
    it's important that this object remains alive while the program is still running.

    [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration
    with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic
    the DeepSpeed configuration is not modified in any way.

    Args:
        config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict.

    """

    def __init__(self, config_file_or_dict):
        # set global weakref object
        set_hf_deepspeed_config(self)
        dep_version_check("accelerate")
        dep_version_check("deepspeed")


SUMMARY:
**Class Summary: `HfDeepSpeedConfig`**

This class extends `DeepSpeedConfig` with optimized configuration for Hugging Face's DeepSpeed trainer.

**Primary Purpose:** Allows for quick querying of DeepSpeed configuration while ensuring the config object remains alive throughout the program execution.

**Key Implementation Details:**

*   Initializes a new instance with a `config_file_or_dict` parameter.
*   Calls parent class constructor (`DeepSpeedConfig`) after performing version checks for `accelerate` and `deepspeed`.
*   Sets a global weakref of this object in the module's globals for access in areas where the Trainer object is not available.

**Notable Dependencies or Requirements:**

*   Extends `DeepSpeedConfig`
*   Uses (`Trainer`) and its related config subclass (`HfTrainerDeepSpeedConfig`)
*   Requires version checks for `accelerate` and `deepspeed`

This summary provides a clear overview of the class's purpose, key implementation details, and dependencies, making it easier to understand and navigate complex codebases.


key: chroma:document
string_value:
CODE:

class HfTrainerDeepSpeedConfig(HfDeepSpeedConfig):
    """
    The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the
    same lifespan as the latter.
    """

    def __init__(self, config_file_or_dict):
        super().__init__(config_file_or_dict)
        self._dtype = None
        self.mismatches = []

    def dtype(self):
        if self._dtype is None:
            raise ValueError("trainer_config_process() wasn't called yet to tell dtype")
        return self._dtype

    def is_auto(self, ds_key_long):
        val = self.get_value(ds_key_long)
        if val is None:
            return False
        else:
            return val == "auto"

    def fill_match(self, ds_key_long, hf_val, hf_key=None, must_match=True):
        """
        A utility method that massages the config file and can optionally verify that the values match.

        1. Replace "auto" values with `TrainingArguments` value.

        2. If it wasn't "auto" and `must_match` is true, then check that DS config matches Trainer
        config values and if mismatched add the entry to `self.mismatched` - will assert during
        `trainer_config_finalize` for one or more mismatches.

        """
        config, ds_key = self.find_config_node(ds_key_long)
        if config is None:
            return

        if config.get(ds_key) == "auto":
            config[ds_key] = hf_val
            return

        if not must_match:
            return

        ds_val = config.get(ds_key)
        if ds_val is not None and ds_val != hf_val:
            self.mismatches.append(f"- ds {ds_key_long}={ds_val} vs hf {hf_key}={hf_val}")

    fill_only = partialmethod(fill_match, must_match=False)

    def trainer_config_process(self, args, auto_find_batch_size=False):
        """
        Adjust the config with `TrainingArguments` values. This stage is run during `TrainingArguments` object
        creation.
        """
        # DeepSpeed does:
        # train_batch_size = world_size * train_micro_batch_size_per_gpu * gradient_accumulation_steps
        train_batch_size = args.world_size * args.per_device_train_batch_size * args.gradient_accumulation_steps
        self.fill_match(
            "train_micro_batch_size_per_gpu",
            args.per_device_train_batch_size,
            "per_device_train_batch_size",
            not auto_find_batch_size,
        )
        self.fill_match(
            "gradient_accumulation_steps",
            args.gradient_accumulation_steps,
            "gradient_accumulation_steps",
        )
        self.fill_match(
            "train_batch_size",
            train_batch_size,
            "train_batch_size (calculated)",
            not auto_find_batch_size,
        )
        self.fill_match("gradient_clipping", args.max_grad_norm, "max_grad_norm")

        self.fill_match("optimizer.params.lr", args.learning_rate, "learning_rate")
        self.fill_match(
            "optimizer.params.betas",
            [args.adam_beta1, args.adam_beta2],
            "adam_beta1+adam_beta2",
        )
        self.fill_match("optimizer.params.eps", args.adam_epsilon, "adam_epsilon")
        self.fill_match("optimizer.params.weight_decay", args.weight_decay, "weight_decay")

        self.fill_only("scheduler.params.warmup_min_lr", 0)  # not a trainer arg
        self.fill_match("scheduler.params.warmup_max_lr", args.learning_rate, "learning_rate")
        # total_num_steps - will get set in trainer_config_finalize

        # fp16
        if args.fp16 or args.fp16_full_eval:
            fp16_backend = "apex" if args.fp16_backend == "apex" else "amp"
        else:
            fp16_backend = None

        if args.save_on_each_node:
            # deepspeed uses shared storage by default. Let's override this setting if save_on_each_node == True
            self.config["checkpoint"] = self.config.get("checkpoint", {})
            self.config["checkpoint"]["use_node_local_storage"] = args.save_on_each_node

        # amp: similar to the pytorch native amp - it has a bunch of optional params but we won't set
        # any here unless the user did the work
        self.fill_match(
            "fp16.enabled",
            ((args.fp16 or args.fp16_full_eval) and fp16_backend == "amp"),
            "fp16|fp16_full_eval+fp16_backend(amp)",
        )

        # apex: delegates amp work to apex (which needs to be available), but it cannot be used with any
        # ZeRO features
        self.fill_match("amp.enabled", fp16_backend == "apex", "fp16+fp16_backend(apex)")
        self.fill_match("amp.opt_level", args.fp16_opt_level, "fp16_opt_level")

        self.fill_match("bf16.enabled", (args.bf16 or args.bf16_full_eval), "bf16|bf16_full_eval")

        # deepspeed's default mode is fp16 unless there is a config that says differently
        if self.is_true("bf16.enabled"):
            self._dtype = torch.bfloat16
        elif self.is_false("fp16.enabled"):
            self._dtype = torch.float32
        else:
            self._dtype = torch.float16

    def trainer_config_finalize(self, args, model, num_training_steps):
        """
        This stage is run after we have the model and know num_training_steps.

        Now we can complete the configuration process.
        """
        # zero

        # deal with config keys that use `auto` value and rely on model's hidden_size
        hidden_size_based_keys = [
            "zero_optimization.reduce_bucket_size",
            "zero_optimization.stage3_prefetch_bucket_size",
            "zero_optimization.stage3_param_persistence_threshold",
        ]
        hidden_size_auto_keys = [x for x in hidden_size_based_keys if self.is_auto(x)]

        if len(hidden_size_auto_keys) > 0:
            if hasattr(model.config, "hidden_size"):
                hidden_size = model.config.hidden_size
            elif hasattr(model.config, "hidden_sizes"):
                # if there are many hidden sizes pick the largest one
                hidden_size = max(model.config.hidden_sizes)
            elif hasattr(model.config, "text_config") and hasattr(model.config.text_config, "hidden_size"):
                hidden_size = model.config.text_config.hidden_size
            elif hasattr(model.config, "text_config") and hasattr(model.config.text_config, "hidden_sizes"):
                # if there are many hidden sizes pick the largest one
                hidden_size = max(model.config.text_config.hidden_sizes)
            else:
                raise ValueError(
                    "The model's config file has neither `hidden_size` nor `hidden_sizes` entry, "
                    "therefore it's not possible to automatically fill out the following `auto` entries "
                    f"in the DeepSpeed config file: {hidden_size_auto_keys}. You can fix that by replacing "
                    "`auto` values for these keys with an integer value of your choice."
                )

            self.fill_only("zero_optimization.reduce_bucket_size", hidden_size * hidden_size)
            if self.is_zero3():
                # automatically assign the optimal config values based on model config
                self.fill_only(
                    "zero_optimization.stage3_prefetch_bucket_size",
                    int(0.9 * hidden_size * hidden_size),
                )
                self.fill_only(
                    "zero_optimization.stage3_param_persistence_threshold",
                    10 * hidden_size,
                )

        # scheduler
        self.fill_match(
            "scheduler.params.total_num_steps",
            num_training_steps,
            "num_training_steps (calculated)",
        )
        self.fill_match(
            "scheduler.params.warmup_num_steps",
            args.get_warmup_steps(num_training_steps),
            "warmup_steps",
        )

        if len(self.mismatches) > 0:
            mismatches = "\n".join(self.mismatches)
            raise ValueError(
                "Please correct the following DeepSpeed config values that mismatch TrainingArguments"
                f" values:\n{mismatches}\nThe easiest method is to set these DeepSpeed config values to 'auto'."


SUMMARY:
# HfTrainerDeepSpeedConfig Class Summary

## Purpose and Functionality
The `HfTrainerDeepSpeedConfig` class is a subclass of `HfDeepSpeedConfig`, used for configuring DeepSpeed-based training in Hugging Face Transformers. It adjusts the configuration with TrainingArguments values, ensuring consistency between Trainer and DS config.

## Key Implementation Details

*   Initializes configuration file or dictionary.
*   Processes `TrainingArguments` values to fill in missing DS config keys based on their corresponding Trainer config values.
*   Adjusts `fp16`, `amp`, and `bf16` settings according to Trainer config values.
*   Assigns default data type (`torch.float16`, `torch.bfloat16`) or uses Trainer-provided values depending on `fp16.enabled`.
*   Fills in `Scheduler` parameters based on Trainer's configuration.

## Notes
The class contains error handling mechanisms, such as raised exceptions when missing config values are encountered. It also checks for auto-fillable keys and raises a ValueError if any mismatch is detected that cannot be resolved with default or trainable settings.

### Configuration Data Types

-   `torch.float16`: Configured to use `fp16` features unless overridden by other data types.
-   `torch.bfloat16`: Used if `bf16.enabled` overrides all other data types.


key: chroma:document
string_value:
CODE:

def set_hf_deepspeed_config(hf_deepspeed_config_obj):
    # this is a special weakref global object to allow us to get to Deepspeed config from APIs
    # that don't have an easy way to get to the Deepspeed config outside of the Trainer domain.
    global _hf_deepspeed_config_weak_ref
    # will go away automatically when HfDeepSpeedConfig is destroyed (when TrainingArguments is destroyed)


SUMMARY:
**Function Summary**

`set_hf_deepspeed_config` sets up a global weak reference to hold the `HfDeepSpeedConfig` object, allowing it to be accessed across domains.

**Key Implementation Details**

* Uses a weak reference (`_hf_deepspeed_config_weak_ref`) to avoid circular references.
* Set up weak reference globally for domain-agnostic access.

**Notable Dependencies/Requirements**

* `_HF_DEEPSPEED_CONFIG_WIDTH_REF` is expected but not set, indicating an external global variable.
* Inherently uses Hugging Face Training and Deepspeed libraries.


key: chroma:document
string_value:
CODE:

def unset_hf_deepspeed_config():
    # useful for unit tests to ensure the global state doesn't leak - call from `tearDown` method
    global _hf_deepspeed_config_weak_ref


SUMMARY:
**Code Summary:**

**Function:** unset_hf_deepspeed_config()
**Purpose:** Resets the global internal weak reference to prevent potential memory leaks.

**Key Implementation Details:**
- Sets `_hf_deepspeed_config_weak_ref` to `None`

**Notable Dependencies/Requirements:**
- Uses a global variable `_hf_deepspeed_config_weak_ref`
- Assumes availability of related classes/functions as context.


key: chroma:document
string_value:
CODE:

def is_deepspeed_zero3_enabled():
    if _hf_deepspeed_config_weak_ref is not None and _hf_deepspeed_config_weak_ref() is not None:
        return _hf_deepspeed_config_weak_ref().is_zero3()
    else:


SUMMARY:
**Summary:**

The `_is_deepspeed_zero3_enabled()` function checks if DeepSpeed Zero3 is enabled for the Hugging Face configuration. It returns True if the weak reference to the config object exists and holds an instance of `Zero3`, and False otherwise.

**Key Details:**

* Function signature: `_is_deepspeed_zero3_enabled()`
* Returns a boolean indicating whether DeepSpeed Zero3 is enabled

**Implementation Notes:**

* Uses a weak reference (`_hf_deepspeed_config_weak_ref`) to manage the config object's lifecycle.
* Calls the `is_zero3()` method on the config object to determine its version.


key: chroma:document
string_value:
CODE:

def deepspeed_config():
    if _hf_deepspeed_config_weak_ref is not None and _hf_deepspeed_config_weak_ref() is not None:
        return _hf_deepspeed_config_weak_ref().config
    else:


SUMMARY:
**Summary:**

This function retrieves DeepSpeed configuration from either a weak reference (`_hf_deepspeed_config_weak_ref`) or returns `None` if unavailable.

**Key Implementation Details:**

* Checks for existence of `_hf_deepspeed_config_weak_ref` and evaluates it to `None`
* If available, returns the value of `_hf_deepspeed_config_weak_real()`.
* Method signature: `deepspeed_config`

**Notable Dependencies/Requirements:**

* Inherited interface from `_hf_deepspeed_config_weak_real` (exact implementation not shown in this chunk)
* External dependency: None noted. However, this code snippet suggests a connection to `_hf_deepspeed_config_weak_ref()`, possibly implying an external library or module named `deepsspeed`.


key: chroma:document
string_value:
CODE:

def deepspeed_optim_sched(trainer, hf_deepspeed_config, args, num_training_steps, model_parameters):
    """
    A convenience wrapper that deals with optimizer and lr scheduler configuration.
    """
    from accelerate.utils import DummyOptim, DummyScheduler

    config = hf_deepspeed_config.config

    # Mixing and matching DS schedulers and optimizers is supported unless Offload is enabled in which case it's:
    # 1. DS scheduler + DS optimizer: Yes
    # 2. HF scheduler + HF optimizer: Mostly*
    # 3. DS scheduler + HF optimizer: Mostly*
    # 4. HF scheduler + DS optimizer: Yes
    #
    # Mostly*: All non-native DeepSpeed optimizers that have both CPU and GPU implementation should work (except LAMB)

    optimizer = None
    if "optimizer" in config:
        if args.adafactor:
            raise ValueError(
                "--adafactor was passed, but also found `optimizer` configured in the DeepSpeed config. "
                "Only one optimizer can be configured."
            )
        optimizer = DummyOptim(params=model_parameters)
    else:
        if hf_deepspeed_config.is_offload():
            logger.info(
                "Detected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the"
                " custom optimizer has both CPU and GPU implementation (except LAMB)"
            )

        # ds supports Adam, OneBitAdam, and Lamb optimizers and can import other optimizers from torch.
        # But trainer uses AdamW by default.
        optimizer = trainer.create_optimizer()
        # To use other optimizers requires voiding warranty with: `zero_allow_untested_optimizer`
        config["zero_allow_untested_optimizer"] = True

    lr_scheduler = None
    if "scheduler" in config:
        lr_scheduler = DummyScheduler(optimizer)
    else:
        if isinstance(optimizer, DummyOptim):

            def _lr_scheduler_callable(optimizer):
                # create a shallow copy first, so later modifications do not affect original trainer
                trainer_copy = copy.copy(trainer)
                # at the time _lr_scheduler_callable is called, trainer.lr_scheduler has been set
                # update it to None so that we can re-create a new scheduler
                trainer_copy.lr_scheduler = None
                lr_scheduler = trainer_copy.create_scheduler(
                    num_training_steps=num_training_steps, optimizer=optimizer
                )
                return lr_scheduler

            lr_scheduler = DummyScheduler(optimizer, lr_scheduler_callable=_lr_scheduler_callable)
        else:
            lr_scheduler = trainer.create_scheduler(num_training_steps=num_training_steps, optimizer=optimizer)



SUMMARY:
**DeepSpeed Optimizer Schedulizer Function Summary**

**Main Purpose and Functionality:**
This function, `deepspeed_optim_sched`, is a convenience wrapper that configures the optimizer and learning rate scheduler for deep speed training. It handles mixing and matching of DS schedulers and optimizers, as well as configuring them accordingly.

**Key Implementation Details:**

* Creates an instance of `DummyOptim` with model parameters (`model_parameters`) if `optimizer` is not specified in `hf_deepspeed_config`.
* Configures non-DeepSpeed optimizer (e.g., AdamW) using `trainer.create_optimizer()`.
* Setups `lr_scheduler` by either creating a new instance of `DummyScheduler` or copying the original trainer's scheduler.
* Handles cases where `--adafactor` is passed and optimizers are mixed with DeepSpeed.

**Notable Dependencies and Requirements:**

* `accelerate.utils`: `DummyOptim`, `DummyScheduler`
* `torch`: for importing custom optimizers
* `trainer` object from `accelerate trainers library`: used for creating optimizers and schedulers

**Return Values:**
The function returns a tuple containing the configured optimizer (`optimizer`) and learning rate scheduler (`lr_scheduler`).


key: chroma:document
string_value:
CODE:

def deepspeed_init(trainer, num_training_steps, inference=False):
    """
    Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args.

    If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made.

    Args:
        trainer: Trainer object
        num_training_steps: per single gpu
        resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load
        inference: launch in inference mode (no optimizer and no lr scheduler)
        auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being
            set automatically by the auto batch size finder

    Returns: optimizer, lr_scheduler

    We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on:
    https://github.com/microsoft/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it
    can't resume from a checkpoint after it did some stepping https://github.com/microsoft/DeepSpeed/issues/1612

    """
    from deepspeed.utils import logger as ds_logger

    model = trainer.model
    args = trainer.args

    hf_deepspeed_config = trainer.accelerator.state.deepspeed_plugin.hf_ds_config

    # resume config update - some bits like `model` and `num_training_steps` only become available during train
    hf_deepspeed_config.trainer_config_finalize(args, model, num_training_steps)

    # set the Deepspeed log level consistent with the Trainer
    ds_logger.setLevel(args.get_process_log_level())

    if inference:
        # only Z3 makes sense for the inference
        if not hf_deepspeed_config.is_zero3():
            raise ValueError("ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config")

        # in case the training config is re-used for inference
        hf_deepspeed_config.del_config_sub_tree("optimizer")
        hf_deepspeed_config.del_config_sub_tree("lr_scheduler")
        optimizer, lr_scheduler = None, None
        model_parameters = None
    else:
        trainer.optimizer = None  # important for when deepspeed_init is used as re-init
        model_parameters = list(filter(lambda p: p.requires_grad, model.parameters()))
        optimizer, lr_scheduler = deepspeed_optim_sched(
            trainer, hf_deepspeed_config, args, num_training_steps, model_parameters
        )

    # keep for quick debug:
    # from pprint import pprint; pprint(config)



SUMMARY:
**Function Summary:**

`deepspeed_init(trainer, num_training_steps, inference=False)`

Initiates DeepSpeed with the given `trainer` and configures it for training. This function updates the DeepSpeed configuration and initializes the optimizer and learning rate scheduler accordingly.

**Key Implementation Details:**

* Initializes Deepspeed logger based on trainer's log level.
* Adapts to inference mode by clearing optimizer and LR scheduler configurations.
* Configures optimizer and LR scheduler using `deepspeed_optim_sched` function.

**Notable Dependencies and Requirements:**

* `trainer.accelerator.state.deepspeed_plugin.hf_ds_config`
* `torch.optim` (optimizer) and `torch.optim.lr_scheduler` (LR scheduler)

**Return Values:**

* `optimizer`: initialized DeepSpeed optimizer.
* `lr_scheduler`: initialized DeepSpeed learning rate scheduler.


key: chroma:document
string_value:
CODE:

def deepspeed_load_checkpoint(deepspeed_engine, checkpoint_path, load_module_strict=True):
    # it's possible that the user is trying to resume from model_path, which doesn't necessarily
    # contain a deepspeed checkpoint. e.g. examples just check if the dir exists and assume it's
    # a resume from a checkpoint and not just a local pretrained weight. So we check here if the
    # path contains what looks like a deepspeed checkpoint
    import glob

    deepspeed_checkpoint_dirs = sorted(glob.glob(f"{checkpoint_path}/global_step*"))

    if len(deepspeed_checkpoint_dirs) > 0:
        logger.info(f"Attempting to resume from {checkpoint_path}")
        # this magically updates self.optimizer and self.lr_scheduler
        load_path, _ = deepspeed_engine.load_checkpoint(
            checkpoint_path,
            load_module_strict=load_module_strict,
            load_optimizer_states=True,
            load_lr_scheduler_states=True,
        )
        if load_path is None:
            raise ValueError(f"[deepspeed] failed to resume from checkpoint {checkpoint_path}")
    else:


SUMMARY:
**Summary: `deepspeed_load_checkpoint` Function**

**Purpose and Functionality:**
The `deepspeed_load_checkpoint` function loads a checkpoint from a specified path using the DeepSpeed engine. It checks if the path contains a valid DeepSpeed checkpoint by looking for directories with a "global_step*" directory.

**Key Details:**

* Loads optimizer and LR scheduler states automatically
* Uses `glob.glob` to find DeepSpeed checkpoint directories
* Raises `ValueError` exceptions if no valid checkpoint is found or if loading fails

**Notable Dependencies and Interfaces:**
- `[deepspeed]`: Loaded from a specified checkpoint path, with optional `load_module_strict=True`.

**External Dependencies:**

- `[glob]`
- `[deepspeed_engine]`

Note: The code uses both Python-specific terms (e.g., glob) and library-specific names (e.g., [deepspeed_engine]) to maintain exactness.


key: chroma:document
string_value:
CODE:

def set_module_quantized_tensor_to_device(module, tensor_name, device, value=None, quantized_stats=None):
    """
    A helper function to set a given tensor (parameter of buffer) of a module on a specific device (note that doing
    `param.to(device)` creates a new tensor not linked to the parameter, which is why we need this function). The
    function is adapted from `set_module_tensor_to_device` function from accelerate that is adapted to support the
    class `Int8Params` from `bitsandbytes`.

    Args:
        module (`torch.nn.Module`):
            The module in which the tensor we want to move lives.
        tensor_name (`str`):
            The full name of the parameter/buffer.
        device (`int`, `str` or `torch.device`):
            The device on which to set the tensor.
        value (`torch.Tensor`, *optional*):
            The value of the tensor (useful when going from the meta device to any other device).
        quantized_stats (`dict[str, Any]`, *optional*):
            Dict with items for either 4-bit or 8-bit serialization
    """
    # Recurse if needed
    if "." in tensor_name:
        splits = tensor_name.split(".")
        for split in splits[:-1]:
            new_module = getattr(module, split)
            if new_module is None:
                raise ValueError(f"{module} has no attribute {split}.")
            module = new_module
        tensor_name = splits[-1]

    if tensor_name not in module._parameters and tensor_name not in module._buffers:
        raise ValueError(f"{module} does not have a parameter or a buffer named {tensor_name}.")
    is_buffer = tensor_name in module._buffers
    old_value = getattr(module, tensor_name)

    if old_value.device == torch.device("meta") and device not in ["meta", torch.device("meta")] and value is None:
        raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {device}.")

    prequantized_loading = quantized_stats is not None
    if is_buffer or not is_bitsandbytes_available():
        is_8bit = False
        is_4bit = False
    else:
        is_4bit = hasattr(bnb.nn, "Params4bit") and isinstance(module._parameters[tensor_name], bnb.nn.Params4bit)
        is_8bit = isinstance(module._parameters[tensor_name], bnb.nn.Int8Params)

    if is_8bit or is_4bit:
        param = module._parameters[tensor_name]
        if param.device.type != "cuda":
            if value is None:
                new_value = old_value.to(device)
            elif isinstance(value, torch.Tensor):
                new_value = value.to("cpu")
            else:
                new_value = torch.tensor(value, device="cpu")

            # Support models using `Conv1D` in place of `nn.Linear` (e.g. openai-community/gpt2) by transposing the weight matrix prior to quantization.
            # Since weights are saved in the correct "orientation", we skip transposing when loading.
            if issubclass(module.source_cls, Conv1D) and not prequantized_loading:
                new_value = new_value.T

            kwargs = old_value.__dict__

            if prequantized_loading != (new_value.dtype in (torch.int8, torch.uint8)):
                raise ValueError(
                    f"Value dtype `{new_value.dtype}` is not compatible with parameter quantization status."
                )

            if is_8bit:
                is_8bit_serializable = version.parse(importlib.metadata.version("bitsandbytes")) > version.parse(
                    "0.37.2"
                )
                if new_value.dtype in (torch.int8, torch.uint8) and not is_8bit_serializable:
                    raise ValueError(
                        "Detected int8 weights but the version of bitsandbytes is not compatible with int8 serialization. "
                        "Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`."
                    )
                new_value = bnb.nn.Int8Params(new_value, requires_grad=False, **kwargs).to(device)
                if prequantized_loading:
                    setattr(new_value, "SCB", quantized_stats["SCB"].to(device))
            elif is_4bit:
                if prequantized_loading:
                    is_4bit_serializable = version.parse(importlib.metadata.version("bitsandbytes")) >= version.parse(
                        "0.41.3"
                    )
                    if new_value.dtype in (torch.int8, torch.uint8) and not is_4bit_serializable:
                        raise ValueError(
                            "Detected 4-bit weights but the version of bitsandbytes is not compatible with 4-bit serialization. "
                            "Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`."
                        )
                    new_value = bnb.nn.Params4bit.from_prequantized(
                        data=new_value,
                        quantized_stats=quantized_stats,
                        requires_grad=False,
                        device=device,
                        **kwargs,
                    )
                else:
                    new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).to(device)
            module._parameters[tensor_name] = new_value

    else:
        if value is None:
            new_value = old_value.to(device)
        elif isinstance(value, torch.Tensor):
            new_value = value.to(device)
        else:
            new_value = torch.tensor(value, device=device)

        if is_buffer:
            module._buffers[tensor_name] = new_value
        else:
            new_value = nn.Parameter(new_value, requires_grad=old_value.requires_grad)


SUMMARY:
**Summary:**

The `set_module_quantized_tensor_to_device` function is a helper function for setting a tensor on a specific device in a PyTorch model. It adapts the Accelerate library's functionality to support Int8Params from BitsAndBytes.

**Primary Purpose:** Set a tensor on a specified device, taking into account quantization and serialization requirements.

**Key Implementation Details:**

* Handles buffers and parameters separately
* Supports 4-bit and 8-bit serialization and quantization
* Recurses through module hierarchy for nested tensors
* Transposes weight matrix if necessary to support Conv1D models

**Method Signatures and Interfaces:**

* `module._parameters[tensor_name]`: accessed for setting value on device
* `bnb.nn.Int8Params` and `bnb.nn.Params4bit` classes used for quantization
* `torch.device` type for specifying target device

**Notable Dependencies and Requirements:**

* `bitsandbytes` library for Int8Params and Params4bit serialization
* PyTorch library for tensor operations
* Accelerate library's functionality adapted to support BitsAndBytes interfaces.


key: chroma:document
string_value:
CODE:

def _replace_with_bnb_linear(
    model,
    modules_to_not_convert=None,
    current_key_name=None,
    quantization_config=None,
    has_been_replaced=False,
):
    """
    Private method that wraps the recursion for module replacement.

    Returns the converted model and a boolean that indicates if the conversion has been successfull or not.
    """
    for name, module in model.named_children():
        if current_key_name is None:
            current_key_name = []
        current_key_name.append(name)

        if (isinstance(module, nn.Linear) or isinstance(module, Conv1D)) and name not in modules_to_not_convert:
            # Check if the current key is not in the `modules_to_not_convert`
            current_key_name_str = ".".join(current_key_name)
            if not any(
                (key + "." in current_key_name_str) or (key == current_key_name_str) for key in modules_to_not_convert
            ):
                with init_empty_weights():
                    if isinstance(module, Conv1D):
                        in_features, out_features = module.weight.shape
                    else:
                        in_features = module.in_features
                        out_features = module.out_features

                    if quantization_config.quantization_method() == "llm_int8":
                        model._modules[name] = bnb.nn.Linear8bitLt(
                            in_features,
                            out_features,
                            module.bias is not None,
                            has_fp16_weights=quantization_config.llm_int8_has_fp16_weight,
                            threshold=quantization_config.llm_int8_threshold,
                        )
                        has_been_replaced = True
                    else:
                        if (
                            quantization_config.llm_int8_skip_modules is not None
                            and name in quantization_config.llm_int8_skip_modules
                        ):
                            pass
                        else:
                            extra_kwargs = (
                                {"quant_storage": quantization_config.bnb_4bit_quant_storage}
                                if "quant_storage" in list(signature(bnb.nn.Linear4bit).parameters)
                                else {}
                            )
                            model._modules[name] = bnb.nn.Linear4bit(
                                in_features,
                                out_features,
                                module.bias is not None,
                                quantization_config.bnb_4bit_compute_dtype,
                                compress_statistics=quantization_config.bnb_4bit_use_double_quant,
                                quant_type=quantization_config.bnb_4bit_quant_type,
                                **extra_kwargs,
                            )
                            has_been_replaced = True
                    # Store the module class in case we need to transpose the weight later
                    model._modules[name].source_cls = type(module)
                    # Force requires grad to False to avoid unexpected errors
                    model._modules[name].requires_grad_(False)
        if len(list(module.children())) > 0:
            _, has_been_replaced = _replace_with_bnb_linear(
                module,
                modules_to_not_convert,
                current_key_name,
                quantization_config,
                has_been_replaced=has_been_replaced,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Summary:**

This function `_replace_with_bnb_linear` replaces linear and Conv1D modules in a PyTorch model with BNB (Binary Nested Breakdown) variants. It recursively traverses the model's nested structures, replacing specified modules while preserving others.

**Key Implementation Details:**

*   Replaces `nn.Linear` and `Conv1D` modules based on config.
*   Handles skip modules for specific quantization methods ("llm_int8").
*   Employs BNB variants (`Linear4bit`, `Linear8bitLt`) with custom configuration.
*   Iteratively replaces module weights, biases, and computes gradients in-place.

**Notable Dependencies and Requirements:**

*   PyTorch (specifically `nn.Linear` and `Conv1D`)
*   `bnb` library for BNB variants (`Linear4bit`, `Linear8bitLt`)
*   Optional dependencies:
    *   TorchScript or TorchX for computation optimization
        *   Requires `init_empty_weights()` and has FP16 weights.
    *   Quantization config and dependencies within the `quantization_config` object.


key: chroma:document
string_value:
CODE:

def replace_with_bnb_linear(model, modules_to_not_convert=None, current_key_name=None, quantization_config=None):
    """
    A helper function to replace all `torch.nn.Linear` modules by `bnb.nn.Linear8bit` modules from the `bitsandbytes`
    library. This will enable running your models using mixed int8 precision as described by the paper `LLM.int8():
    8-bit Matrix Multiplication for Transformers at Scale`. Make sure `bitsandbytes` compiled with the correct CUDA
    version of your hardware is installed before running this function. `pip install -i https://test.pypi.org/simple/
    bitsandbytes`

    The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should
    be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no
    CPU/GPU memory is required to run this function. Int8 mixed-precision matrix decomposition works by separating a
    matrix multiplication into two streams: (1) and systematic feature outlier stream matrix multiplied in fp16
    (0.01%), (2) a regular stream of int8 matrix multiplication (99.9%). With this method, int8 inference with no
    predictive degradation is possible for very large models (>=176B parameters).

    Parameters:
        model (`torch.nn.Module`):
            Input model or `torch.nn.Module` as the function is run recursively.
        modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`):
            Names of the modules to not convert in `Linear8bitLt`. In practice we keep the `lm_head` in full precision
            for numerical stability reasons.
        current_key_name (`List[`str`]`, *optional*):
            An array to track the current key of the recursion. This is used to check whether the current key (part of
            it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or
            `disk`).
        quantization_config ('transformers.utils.quantization_config.BitsAndBytesConfig'):
            To configure and manage settings related to quantization, a technique used to compress neural network models
            by reducing the precision of the weights and activations, thus making models more efficient in terms of both
            storage and computation.
    """
    modules_to_not_convert = ["lm_head"] if modules_to_not_convert is None else modules_to_not_convert
    model, has_been_replaced = _replace_with_bnb_linear(
        model, modules_to_not_convert, current_key_name, quantization_config
    )

    if not has_been_replaced:
        logger.warning(
            "You are loading your model in 8bit or 4bit but no linear modules were found in your model."
            " Please double check your model architecture, or submit an issue on github if you think this is"
            " a bug."
        )



SUMMARY:
**Summary:**

`replace_with_bnb_linear` function is a recursive helper function to replace all `torch.nn.Linear` modules in a PyTorch model with `bnb.nn.Linear8bit` modules from the `bitsandbytes` library. The replacement excludes specific modules specified by `modules_to_not_convert`. This enables mixed int8 precision for efficient inference.

**Key Details:**

* Replaces `torch.nn.Linear` with `bnb.nn.Linear8bit`
* Excludes `lm_head` and other `modules_to_not_convert` exceptions
* Implemented under `init_empty_weights` context manager to avoid memory usage
* Employs int8 mixed-precision matrix decomposition for efficient inference

**Notable Dependencies:**

* `bitsandbytes` library (compiled with correct CUDA version)
* `torch` library
* `transformers.utils.quantization_config.BitsAndBytesConfig`

**Method Signatures and Interfaces:**

* `_replace_with_bnb_linear(model, modules_to_not_convert, current_key_name, quantization_config)`

Note: The original code's comments are preserved in the summary for better understanding.


key: chroma:document
string_value:
CODE:
# For backward compatibility
def replace_8bit_linear(*args, **kwargs):
    warnings.warn(
        "`replace_8bit_linear` will be deprecated in a future version, please use `replace_with_bnb_linear` instead",
        FutureWarning,
    )


SUMMARY:
**Deprecated Function Summary**

`replace_8bit_linear`

**Redeprecated** (in favor of `replace_with_bnb_linear`)

Replaces 8-bit linear scaling with BNB-linear scaling.

**Key Implementation Details:**

* Calls deprecated warning and redirects to newer function `replace_with_bnb_linear`
* Notifies user of deprecation notice

**Notable Dependencies/Requirements:**

* None explicitly referenced
* Implications: Dependent on deprecated module or library


key: chroma:document
string_value:
CODE:
# For backward compatiblity
def set_module_8bit_tensor_to_device(*args, **kwargs):
    warnings.warn(
        "`set_module_8bit_tensor_to_device` will be deprecated in a future version, please use `set_module_quantized_tensor_to_device` instead",
        FutureWarning,
    )


SUMMARY:
**Function Summary**

```
set_module_8bit_tensor_to_device(*args, **kwargs):
  Deprecation Warning: Redirects to set_module_quantized_tensor_to_device with updated arguments.
  Replaces deprecated functionality in future versions.
```

**Key Implementation Details**

- Replaces `set_module_8bit_tensor_to_device` with the modern and compatible version `set_module_quantized_tensor_to_device`.
- Provides a deprecation warning, replacing the function with the future-friendly alternative.

**Notable Dependencies or Requirements**

*   `FutureWarning`
    -   This was used as a warning type to signal future deprecation.


key: chroma:document
string_value:
CODE:

def get_keys_to_not_convert(model):
    r"""
    An utility function to get the key of the module to keep in full precision if any For example for CausalLM modules
    we may want to keep the lm_head in full precision for numerical stability reasons. For other architectures, we want
    to keep the tied weights of the model. The function will return a list of the keys of the modules to not convert in
    int8.

    Parameters:
    model (`torch.nn.Module`):
        Input model
    """
    # Create a copy of the model and tie the weights, then
    # check if it contains tied weights
    tied_model = deepcopy(model)  # this has 0 cost since it is done inside `init_empty_weights` context manager`
    tied_model.tie_weights()

    tied_params = find_tied_parameters(tied_model)
    # For compatibility with Accelerate < 0.18
    if isinstance(tied_params, dict):
        tied_keys = sum(list(tied_params.values()), []) + list(tied_params.keys())
    else:
        tied_keys = sum(tied_params, [])
    has_tied_params = len(tied_keys) > 0

    # If there is not tied weights, we want to keep the lm_head（output_embedding) in full precision
    if not has_tied_params:
        output_emb = model.get_output_embeddings()
        if output_emb is not None:
            list_last_module = [name for name, module in model.named_modules() if id(module) == id(output_emb)]
            return list_last_module

    # otherwise, no tied weights, no output embedding defined, simply keep the last module in full precision
    list_modules = list(model.named_parameters())
    list_last_module = [list_modules[-1][0]]
    # add last module together with tied weights
    intersection = set(list_last_module) - set(tied_keys)
    list_untouched = list(set(tied_keys)) + list(intersection)

    # remove ".weight" from the keys
    names_to_remove = [".weight", ".bias"]
    filtered_module_names = []
    for name in list_untouched:
        for name_to_remove in names_to_remove:
            if name_to_remove in name:
                name = name.replace(name_to_remove, "")
        filtered_module_names.append(name)



SUMMARY:
**Summary:**

This function `get_keys_to_not_convert` is used to determine the keys of a PyTorch model that should not be converted to int8 precision. The function takes into account specific model architectures, such as CausalLM modules with tied weights, and returns a list of module names.

**Key Functionality:**

1. Ties weights in the input model using `tie_weights()`.
2. Finds tied parameters using `find_tied_parameters()` and extracts their keys.
3. Checks for compatibility with Accelerate < 0.18 version.
4. If no tied weights are present, checks if an output embedding is defined and keeps the last module in full precision otherwise keeps the last module.

**Key Implementation Details:**

* Uses `deepcopy` to create a copy of the input model without additional cost.
* Utilizes the `named_modules()` method to iterate through model layers.
* Applies filtering to module names by removing ".weight" and ".bias" suffixes.

**Notable Dependencies/Requirements:**

* PyTorch library (`torch.nn.Module`)
* Accelerate library (specifically used incompatibility fix)
* Custom implementations of `find_tied_parameters()` function.


key: docstring
string_value:
# Copied from PEFT: https://github.com/huggingface/peft/blob/47b3712898539569c02ec5b3ed4a6c36811331a1/src/peft/utils/integrations.py#L41


key: chroma:document
string_value:
CODE:
# Copied from PEFT: https://github.com/huggingface/peft/blob/47b3712898539569c02ec5b3ed4a6c36811331a1/src/peft/utils/integrations.py#L41
def dequantize_bnb_weight(weight: "torch.nn.Parameter", dtype: "torch.dtype", state=None):
    """
    Helper function to dequantize 4bit or 8bit bnb weights.

    If the weight is not a bnb quantized weight, it will be returned as is.
    """
    if not isinstance(weight, torch.nn.Parameter):
        raise TypeError(f"Input weight should be of type nn.Parameter, got {type(weight)} instead")

    cls_name = weight.__class__.__name__
    if cls_name not in ("Params4bit", "Int8Params"):
        return weight

    if cls_name == "Params4bit":
        output_tensor = bnb.functional.dequantize_4bit(weight.data, weight.quant_state)
        logger.warning_once(
            f"The model is going to be dequantized in {output_tensor.dtype} - if you want to upcast it to another dtype, make sure to pass the desired dtype when quantizing the model through `bnb_4bit_quant_type` argument of `BitsAndBytesConfig`"
        )
        return output_tensor.to(dtype)

    if state.SCB is None:
        state.SCB = weight.SCB

    if hasattr(bnb.functional, "int8_vectorwise_dequant"):
        # Use bitsandbytes API if available (requires v0.45.0+)
        dequantized = bnb.functional.int8_vectorwise_dequant(weight.data, state.SCB)
    else:
        # Multiply by (scale/127) to dequantize.
        dequantized = weight.data * state.SCB.view(-1, 1) * 7.874015718698502e-3



SUMMARY:
**Summary**

The `dequantize_bnb_weight` function dequantizes 4-bit or 8-bit bbn weights, returning the weight in its original precision. If the input is not bnn quantized, it is returned unchanged.

**Key Implementation Details:**

* Uses `torch.nn.Parameter` as input weight.
* Validates input weight class using `__class__.__name__`, allowing only `Params4bit` and `Int8Params`.
* Calls `bnb.functional.dequantize_4bit` for 4-bit weights, or uses `int8_vectorwise_dequant` if available (v0.45.0+).
* For 8-bit weights, multiplies the input data by `(scale/127)` to dequantize.

**Notable Dependencies and Requirements:**

* `torch.nn.Parameter`
* `bnb` library (with v0.45.0 or later for `int8_vectorwise_dequant`)
* `logger` (for warning messages)

**Function Signature:**

`dequantize_bnb_weight(weight: "torch.nn.Parameter", dtype: "torch.dtype", state=None) -> torch.Tensor`


key: chroma:document
string_value:
CODE:

def _create_accelerate_new_hook(old_hook):
    r"""
    Creates a new hook based on the old hook. Use it only if you know what you are doing !
    This method is a copy of: https://github.com/huggingface/peft/blob/748f7968f3a31ec06a1c2b0328993319ad9a150a/src/peft/utils/other.py#L245
    with some changes
    """
    old_hook_cls = getattr(accelerate.hooks, old_hook.__class__.__name__)
    old_hook_attr = old_hook.__dict__
    filtered_old_hook_attr = {}
    old_hook_init_signature = inspect.signature(old_hook_cls.__init__)
    for k in old_hook_attr.keys():
        if k in old_hook_init_signature.parameters:
            filtered_old_hook_attr[k] = old_hook_attr[k]
    new_hook = old_hook_cls(**filtered_old_hook_attr)


SUMMARY:
**Summary:**

 `_create_accelerate_new_hook` creates a new hook class by filtering the attributes of an existing hook (`old_hook`). The function assumes knowledge of `old_hook`'s initialization parameters and uses them to initialize the new hook, effectively cloning the original hook's functionality.

**Implementation Details:**

* Filters attributes based on initialization signatures.
* Uses `getattr`, `inspect.signature` for attribute access and type checking.

**Notable Dependencies/Requirements:**

* `accelerate.hooks`: External module containing base hook classes.
* `accelerate`, `inspect`: Imported modules, but not explicitly used in the function snippet.


key: chroma:document
string_value:
CODE:

def _dequantize_and_replace(
    model,
    dtype,
    modules_to_not_convert=None,
    current_key_name=None,
    quantization_config=None,
    has_been_replaced=False,
):
    """
    Converts a quantized model into its dequantized original version. The newly converted model will have
    some performance drop compared to the original model before quantization - use it only for specific usecases
    such as QLoRA adapters merging.

    Returns the converted model and a boolean that indicates if the conversion has been successfull or not.
    """
    quant_method = quantization_config.quantization_method()

    target_cls = bnb.nn.Linear8bitLt if quant_method == "llm_int8" else bnb.nn.Linear4bit

    for name, module in model.named_children():
        if current_key_name is None:
            current_key_name = []
        current_key_name.append(name)

        if isinstance(module, target_cls) and name not in modules_to_not_convert:
            # Check if the current key is not in the `modules_to_not_convert`
            current_key_name_str = ".".join(current_key_name)

            if not any(
                (key + "." in current_key_name_str) or (key == current_key_name_str) for key in modules_to_not_convert
            ):
                bias = getattr(module, "bias", None)

                device = module.weight.device
                with init_empty_weights():
                    new_module = torch.nn.Linear(module.in_features, module.out_features, bias=bias is not None)

                if quant_method == "llm_int8":
                    state = module.state
                else:
                    state = None

                new_module.weight = torch.nn.Parameter(dequantize_bnb_weight(module.weight, dtype, state))

                if bias is not None:
                    new_module.bias = bias

                # Create a new hook and attach it in case we use accelerate
                if hasattr(module, "_hf_hook"):
                    old_hook = module._hf_hook
                    new_hook = _create_accelerate_new_hook(old_hook)

                    remove_hook_from_module(module)
                    add_hook_to_module(new_module, new_hook)

                new_module.to(device)
                model._modules[name] = new_module
                has_been_replaced = True
        if len(list(module.children())) > 0:
            _, has_been_replaced = _dequantize_and_replace(
                module,
                dtype,
                modules_to_not_convert,
                current_key_name,
                quantization_config,
                has_been_replaced=has_been_replaced,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Function Summary: `_dequantize_and_replace`**

This function converts a quantized neural network (NN) model back to its dequantized original form with specified precision (e.g., 8-bit or 4-bit).

**Primary Purpose:** Dequantization of an NN model, converting it from quantized to unquantized.

**Key Implementation Details:**

* Uses `bnb.nn.Linear` classes for 8-bit and 4-bit linear layers.
* Recursively searches and dequantizes child modules.
* Handles bias and weight quantization during conversion.

**Noteable Dependencies/Requirements:**

* Use of `torch` library for tensor operations.
* Import of `bnb` module, which likely contains neural network layers (`nn.Linear`) implementation.
* Calls to `_create_accelerate_new_hook`, `remove_hook_from_module`, and `add_hook_to_module` suggest use of the `accelerate` framework for optimized training.

**Return Value:**
The function returns a tuple containing:

1. The dequantized model object.
2. A boolean indicating whether the conversion was successful (`has_been_replaced`).


key: chroma:document
string_value:
CODE:

def dequantize_and_replace(
    model,
    modules_to_not_convert=None,
    quantization_config=None,
):
    model, has_been_replaced = _dequantize_and_replace(
        model,
        model.dtype,
        modules_to_not_convert=modules_to_not_convert,
        quantization_config=quantization_config,
    )

    if not has_been_replaced:
        logger.warning(
            "For some reason the model has not been properly dequantized. You might see unexpected behavior."
        )



SUMMARY:
**Summary:**

`dequantize_and_replace` function:

* Dequantizes and updates a neural network model based on provided quantization configuration
* Optionally skips specified modules from conversion
* Returns updated model with logging warning if dequantization is incomplete

Details:

* Model conversion is performed via internal `_dequantize_and_replace` helper function
* `modules_to_not_convert` parameter allows selective skipping of module conversion in the optimization process
* `quantization_config` determines the quantization strategy applied to the model


key: chroma:document
string_value:
CODE:

def _validate_bnb_multi_backend_availability(raise_exception):
    import bitsandbytes as bnb

    bnb_supported_devices = getattr(bnb, "supported_torch_devices", set())
    available_devices = get_available_devices()

    if available_devices == {"cpu"} and not is_ipex_available():
        from importlib.util import find_spec

        if find_spec("intel_extension_for_pytorch"):
            logger.warning(
                "You have Intel IPEX installed but if you're intending to use it for CPU, it might not have the right version. Be sure to double check that your PyTorch and IPEX installs are compatible."
            )

        available_devices.discard("cpu")  # Only Intel CPU is supported by BNB at the moment

    if not available_devices.intersection(bnb_supported_devices):
        if raise_exception:
            bnb_supported_devices_with_info = set(  # noqa: C401
                '"cpu" (needs an Intel CPU and intel_extension_for_pytorch installed and compatible with the PyTorch version)'
                if device == "cpu"
                else device
                for device in bnb_supported_devices
            )
            err_msg = (
                f"None of the available devices `available_devices = {available_devices or None}` are supported by the bitsandbytes version you have installed: `bnb_supported_devices = {bnb_supported_devices_with_info}`. "
                "Please check the docs to see if the backend you intend to use is available and how to install it: https://huggingface.co/docs/bitsandbytes/main/en/installation#multi-backend"
            )

            logger.error(err_msg)
            raise RuntimeError(err_msg)

        logger.warning("No supported devices found for bitsandbytes multi-backend.")
        return False

    logger.debug("Multi-backend validation successful.")


SUMMARY:
**Function Summary: `_validate_bnb_multi_backend_availability`**

**Purpose:** Validate the availability of devices for the BNB (Bits and Bytes) multi-backend in PyTorch.

**Main Functionality:**

1. Check if the available devices match the supported devices based on the installed PyTorch version.
2. If Intel IPEX is detected but not compatible with PyTorch, log a warning and remove "cpu" from the available devices.
3. If no matches are found, raise an exception or log a warning depending on the `raise_exception` flag.

**Notable Implementation Details:**

* Uses `bitsandbytes` library to get supported devices.
* Checks for Intel IPEX installation compatibility.
* Removes non-supported devices ("cpu") from the available devices set.
* Raisest error with custom message if no matches are found.

**Dependencies and Interfaces:**

* `bbn.supported_torch_devices`: Set of supported PyTorch devices.
* `get_available_devices()`: Function to get available devices.
* `is_ipex_available()`: Function to check if IPEX is available.
* `find_spec("intel_extension_for_pytorch")`: Finds the Intel Extension for PyTorch module.
* `importlib.util.find_spec()`: Module to find imported specifiers.


key: chroma:document
string_value:
CODE:

def _validate_bnb_cuda_backend_availability(raise_exception):
    if not is_torch_available():
        return False

    import torch

    if not torch.cuda.is_available():
        log_msg = (
            "CUDA is required but not available for bitsandbytes. Please consider installing the multi-platform enabled version of bitsandbytes, which is currently a work in progress. "
            "Please check currently supported platforms and installation instructions at https://huggingface.co/docs/bitsandbytes/main/en/installation#multi-backend"
        )
        if raise_exception:
            logger.error(log_msg)
            raise RuntimeError(log_msg)

        logger.warning(log_msg)
        return False

    logger.debug("CUDA backend validation successful.")


SUMMARY:
**Summary: `_validate_bnb_cuda_backend_availability` Function**

*   **Purpose and Functionality:** Verifies the availability of CUDA support for the `torch` library to determine whether it is a required but unavailable backend.
*   **Key Implementation Details:**
    *   Checks if Torch is available, throwing an exception on failure (with configurable raising behavior).
    *   Checks specific CUDA availability.
*   **Notable Dependencies and Requirements:** 
    *   `torch`, requiring at least CUDA support, or exceptions to be raised according to the input parameters.

**Method Signatures and Interfaces:**
- `_validate_bnb_cuda_backend_availability(raise_exception)`

This function returns a boolean value indicating the backend's availability:

- True if CUDA is available
- False otherwise


key: chroma:document
string_value:
CODE:

def validate_bnb_backend_availability(raise_exception=False):
    """
    Validates if the available devices are supported by bitsandbytes, optionally raising an exception if not.
    """
    if not is_bitsandbytes_available():
        if importlib.util.find_spec("bitsandbytes") and version.parse(
            importlib.metadata.version("bitsandbytes")
        ) < version.parse("0.43.1"):
            return _validate_bnb_cuda_backend_availability(raise_exception)
        return False

    if is_bitsandbytes_multi_backend_available():
        return _validate_bnb_multi_backend_availability(raise_exception)


SUMMARY:
**Concise Summary:**

The `validate_bnb_backend_availability` function determines if the available devices are compatible with BitsAndBytes framework, raising an exception if not. It checks:

* Availability of `bitsandbytes`, with version requirement (`> "0.43.1"`).
* Multi-backend availability.
 
If no exceptions and compatibility is found, it returns False.

**Highlighted Details:**

- `_validate_bnb_cuda_backend_availability(raise_exception)`: Custom validation function used in certain cases.
- `is_bitsandbytes_available()`, `is_bitsandbytes_multi_backend_available()`: Functions checking availability of `bitsandbytes` and multi-backend, respectively.


key: chroma:document
string_value:
CODE:

def replace_quantization_scales(model, model_type):
    from awq.modules.act import ScaledActivation

    if model_type not in AWQ_SCALES_MAPPINGS:
        return model
    for name, module in model.named_children():
        act_name = AWQ_SCALES_MAPPINGS[model_type]["act"]
        layer_before_act_name = AWQ_SCALES_MAPPINGS[model_type]["layer_before_act"]
        if name == act_name and hasattr(model, layer_before_act_name):
            layer_before_act = getattr(model, AWQ_SCALES_MAPPINGS[model_type]["layer_before_act"])
            size = layer_before_act.out_features
            scale_like = torch.ones(size)
            model._modules[name] = ScaledActivation(module, scale_like)
        _ = replace_quantization_scales(module, model_type)


SUMMARY:
**Summary**

Replaces quantization scales in a PyTorch neural network model. It recursively traverses the model's children and updates activations that require scaling. If a scaling factor is required for an activation layer (`AWQ_SCALES_MAPPINGS[model_type]["act"]`), it creates a `ScaledActivation` module with a scaled_like tensor and inserts it into the original model's architecture.

**Key Details**

* Function name: `replace_quantization_scales`
* Purpose: Replace quantization scales in a PyTorch neural network model
* Parameters:
	+ `model`: PyTorch neural network model
	+ `model_type`: Type of activation scaling (e.g. "scaled", "layer_scaled")
* Returns: Modified PyTorch neural network model with updated quantization scaling

**Key Implementation Details**

* Uses recursive function call to traverse the entire model hierarchy
* Creates a new `ScaledActivation` module and inserts it into the original model's architecture using `_modules[name]`
* Updates activations that require scaling by creating a `scaled_like` tensor and passing it to the `ScaledActivation` constructor

**Notable Dependencies/Requirements**

* PyTorch neural network framework (`model`)
* Custom module `AWQ_SCALES_MAPPINGS` defining activation scaling mappings for different model types.


key: chroma:document
string_value:
CODE:

def replace_with_awq_linear(
    model,
    modules_to_not_convert=None,
    quantization_config=None,
    current_key_name=None,
    has_been_replaced=False,
) -> bool:
    """
    Public method that recursively replaces the Linear layers of the given model with AWQ quantized layers.
    `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the
    conversion has been successfull or not.

    During the module replacement, we also infer the backend to use through the `quantization_config` object.

    Args:
        model (`torch.nn.Module`):
            The model to convert, can be any `torch.nn.Module` instance.
        quantization_config (`AwqConfig`):
            The quantization config object that contains the quantization parameters.
        modules_to_not_convert (`list`, *optional*):
            A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be
            converted.
        current_key_name (`list`, *optional*):
            A list that contains the current key name. This is used for recursion and should not be passed by the user.
        has_been_replaced (`bool`, *optional*):
            A boolean that indicates if the conversion has been successful or not. This is used for recursion and
            should not be passed by the user.
    """
    if modules_to_not_convert is None:
        modules_to_not_convert = []

    backend = quantization_config.backend

    if not is_auto_awq_available():
        raise ValueError(
            "AWQ (either `autoawq` or `llmawq`) is not available. Please install it with `pip install autoawq` or check out the installation guide in https://github.com/mit-han-lab/llm-awq"
        )

    if backend == AwqBackendPackingMethod.AUTOAWQ:
        if quantization_config.version == AWQLinearVersion.GEMM:
            from awq.modules.linear.gemm import WQLinear_GEMM

            target_cls = WQLinear_GEMM
        elif quantization_config.version == AWQLinearVersion.GEMV:
            from awq.modules.linear.gemv import WQLinear_GEMV

            target_cls = WQLinear_GEMV
        elif quantization_config.version == AWQLinearVersion.EXLLAMA:
            if quantization_config.exllama_config["version"] == ExllamaVersion.ONE:
                from awq.modules.linear.exllama import WQLinear_Exllama

                target_cls = WQLinear_Exllama
            elif quantization_config.exllama_config["version"] == ExllamaVersion.TWO:
                from awq.modules.linear.exllamav2 import WQLinear_ExllamaV2

                target_cls = WQLinear_ExllamaV2
            else:
                raise ValueError(f"Unrecognized Exllama version: {quantization_config.exllama_config['version']}")
        elif quantization_config.version == AWQLinearVersion.IPEX:
            from awq.modules.linear.gemm_ipex import WQLinear_IPEX

            target_cls = WQLinear_IPEX
        else:
            raise ValueError(f"Unrecognized AWQ version: {quantization_config.version}")
    else:
        from awq.quantize.qmodule import WQLinear

        target_cls = WQLinear

    for name, module in model.named_children():
        if current_key_name is None:
            current_key_name = []
        current_key_name.append(name)

        if isinstance(module, nn.Linear) and name not in modules_to_not_convert:
            # Check if the current key is not in the `modules_to_not_convert`
            if not any(key in ".".join(current_key_name) for key in modules_to_not_convert):
                in_features = module.in_features
                out_features = module.out_features

                model._modules[name] = target_cls(
                    w_bit=quantization_config.bits,
                    group_size=quantization_config.group_size,
                    in_features=in_features,
                    out_features=out_features,
                    bias=module.bias is not None,
                    dev=module.weight.device,
                )
                has_been_replaced = True

                # Force requires grad to False to avoid unexpected errors
                model._modules[name].requires_grad_(False)
        if len(list(module.children())) > 0:
            _, has_been_replaced = replace_with_awq_linear(
                module,
                modules_to_not_convert=modules_to_not_convert,
                current_key_name=current_key_name,
                quantization_config=quantization_config,
                has_been_replaced=has_been_replaced,
            )
        # Remove the last key for recursion
        current_key_name.pop(-1)


SUMMARY:
**Summary**

`replace_with_awq_linear`: Recursively replaces Linear layers in a given PyTorch model with AWQ quantized layers. The method takes into account various modules to not convert and returns the converted model along with a boolean indicating successful conversion.

**Key Implementation Details:**

*   The method uses a recursive approach, replacing each module and its children.
*   It employs a `quantization_config` object to infer the backend used for quantization.
*   Specific Linear module classes are imported based on the `quantization_config.version`.

**Method Signatures and Interfaces:**

*   `WQLinear_GEMM`, `WQLinear_GEMV`, `WQLinear_Exllama`, `WQLinear_IPEX` (imported linear layer classes)
    *   `WQLinear`, `AwqConfig`, `nn.Linear`

**External Dependencies and Requirements:**

*   `pip install autoawq`
*   `PyTorch` library

**Contextualization:**
The method assumes the presence of an `accelerate` module to utilize the AWQ quantization functionality.


key: chroma:document
string_value:
CODE:

def get_modules_to_fuse(model, quantization_config):
    """
    Returns the fusing mapping given the quantization config and the model

    Args:
        model (`~PreTrainedModel`):
            The model to fuse - note this model should have been converted into AWQ format beforehand.
        quantization_config (`~transformers.quantization_config.AWQConfig`):
            The quantization configuration to use.
    """
    if not isinstance(model, PreTrainedModel):
        raise TypeError(f"The model should be an instance of `PreTrainedModel`, got {model.__class__.__name__}")

    # Always default to `quantization_config.modules_to_fuse`
    if quantization_config.modules_to_fuse is not None:
        current_fused_mapping = quantization_config.modules_to_fuse
        current_fused_mapping["max_seq_len"] = quantization_config.fuse_max_seq_len
    elif model.config.model_type in AWQ_FUSED_MAPPINGS:
        current_fused_mapping = AWQ_FUSED_MAPPINGS[model.config.model_type]

        # Properly deal with the case where we have a multi-modal model as well (e.g. Llava)
        config = model.config.get_text_config(decoder=True)

        # Handle hidden_size, num_attention_heads, num_key_value_heads on our own.
        hidden_size = config.hidden_size
        num_attention_heads = config.num_attention_heads
        num_key_value_heads = getattr(config, "num_key_value_heads", num_attention_heads)

        # Fill `current_fused_mapping` with the expected values
        current_fused_mapping["hidden_size"] = hidden_size
        current_fused_mapping["num_attention_heads"] = num_attention_heads
        current_fused_mapping["num_key_value_heads"] = num_key_value_heads
        current_fused_mapping["max_seq_len"] = quantization_config.fuse_max_seq_len
    else:
        raise ValueError(
            "Fusing mapping not found either on the quantization config or the supported `AWQ_FUSED_MAPPINGS`. Please pass a `fused_mapping` argument"
            " in the `quantization_config` or raise an issue on transformers https://github.com/huggingface/transformers to add its support."
        )


SUMMARY:
**get_modules_to_fuse Function Summary**

### Purpose and Functionality

The `get_modules_to_fuse` function returns the fusing mapping for a given model and quantization configuration. It takes into account various factors such as the model type, hidden size, number of attention heads, and maximum sequence length.

### Key Implementation Details

* Default to `quantization_config.modules_to_fuse` if provided.
* For unsupported models, use `AWQ_FUSED_MAPPINGS` based on the model type.
* Handle multi-modal models by extracting relevant configuration values (e.g., hidden size, attention heads).
* Populate `current_fused_mapping` with expected values.

### Notable Dependencies and Requirements

* `PreTrainedModel`
* `transformers.quantization_config.AWQConfig`
* Internal use of `AWQ_FUSED_MAPPINGS` dictionary.


key: chroma:document
string_value:
CODE:

def fuse_awq_modules(model, quantization_config):
    """
    Optionally fuse some modules in the model to speedup inference.

    Args:
        model (`~PreTrainedModel`):
            The model to fuse - note this model should have been converted into AWQ format beforehand.
        quantization_config (`Union[AwqConfig, dict]`):
            The quantization configuration to use.
    """
    # We need to convert it from dict in order to get an AwqConfig object
    # otherwise the fields `backend` etc. will not be available
    # https://github.com/huggingface/transformers/pull/27411#discussion_r1414044495
    if isinstance(quantization_config, dict):
        quantization_config = AwqConfig.from_dict(quantization_config)
    backend = quantization_config.backend

    modules_to_fuse = get_modules_to_fuse(model, quantization_config)
    modules_to_not_convert = getattr(quantization_config, "modules_to_not_convert", None)

    if backend == AwqBackendPackingMethod.AUTOAWQ:
        from awq.modules.fused.attn import QuantAttentionFused
        from awq.modules.fused.mlp import QuantFusedMLP
        from awq.modules.fused.norm import FasterTransformerRMSNorm
    else:
        raise ValueError("Fusing is only supported for the AutoAWQ backend")

    fused_attention_modules = []

    for name, module in model.named_modules():
        if modules_to_not_convert is not None:
            if any(module_name_to_not_convert in name for module_name_to_not_convert in modules_to_not_convert):
                continue

        # Replace layer norms
        _fuse_awq_layernorm(modules_to_fuse["layernorm"], module, FasterTransformerRMSNorm)

        # Replace MLP layers if awq version is not ipex.
        if quantization_config.version != "ipex":
            _fuse_awq_mlp(model, name, modules_to_fuse["mlp"], module, QuantFusedMLP)
        else:
            logger.info("The IPEX version AWQ does not support fuse mlp for now.")

        # Replace attention layers
        attention_has_been_fused = _fuse_awq_attention_layers(
            model, module, modules_to_fuse, name, QuantAttentionFused
        )

        if attention_has_been_fused:
            fused_attention_modules.append(name.split(".")[0])

    # For AWQ fused + Llama we need to set `config._attn_implementation` = "custom" to avoid unexpected behavior and pass
    # `None` attention mask to the fused attention modules as now the attention mask is dropped by our models and dealt
    # by the `AttentionMaskConverter` module.
    if len(fused_attention_modules) > 0:
        for module_name, module in model.named_modules():
            if any(
                module_name in fused_attention_modules for fused_attention_parent_module in fused_attention_modules
            ):
                if hasattr(module, "config") and hasattr(module.config, "_attn_implementation"):
                    module.config._attn_implementation = "custom"


SUMMARY:
** fuse_awq_modules Function Summary **
======================================

**Purpose**
-----------

The `fuse_awq_modules` function optimizes the configuration of an AWQ (Auto-Quantization) formatted pre-trained model by:
1. Fusing selectively modules to speed up inference.
2. Ensuring compatibility with the chosen quantization backend.

**Primary Purpose: Fusing Module Configurations for Efficient Inference**

**Key Implementation Details:**
------------------------------

* Checks if the `quantization_config` is a dictionary and converts it into an `AwqConfig` object.
* Resolves the quantization backend and corresponding fused module classes (`QuantAttentionFused`, `QuantFusedMLP`, and `FasterTransformerRMSNorm`).
* Iterates over the model's named modules, fusing selectively:
	+ Layer norms using `_fuse_awq_layernorm`.
	+ MLP layers (version-specific) using `_fuse_awq_mlp`.
	+ Attention layers using `_fuse_awq_attention_layers`.

**Notable Dependencies and Requirements:**
-----------------------------------------

* `AwqConfig` from Hugging Face's transformers library.
* External dependency on the `awq` library for quantization functionality.

**Key Interfaces and Method Signatures:**

* `modules_to_fuse["layernorm"]`: Layer normalization module.
* `modules_to_fuse["mlp"]`: MLP layer module.
* `QuantAttentionFused`, `QuantFusedMLP`, and `FasterTransformerRMSNorm` fused module classes.


key: chroma:document
string_value:
CODE:

def _fuse_awq_layernorm(fuse_module_names, module, target_cls):
    """
    Fuse the LayerNorm layers into a target class using autoawq

    Args:
        fuse_module_names (`List[str]`):
            The list of module names to fuse
        module (`nn.Module`):
            The pytorch parent module that has layernorm modules to fuse
        target_cls (`~autoawq.FasterTransformerRMSNorm`):
            The `FasterTransformerRMSNorm` class as it only supports that class
            for now.
    """
    for module_name in fuse_module_names:
        if hasattr(module, module_name):
            old_module = getattr(module, module_name)
            module._modules[module_name] = target_cls(
                old_module.weight,
                old_module.variance_epsilon,
            ).to(old_module.weight.device)


SUMMARY:
**Function Summary: `_fuse_awq_layernorm`**

Fuses LayerNorm layers from PyTorch into the provided `target_cls` using autoawq.

* Takes in `module_names` to fuse, `pytorch_modules`, and `target_cls`.
* Replaces existing LayerNorm modules with `target_cls` instances.
* Modifies `module._modules` to store fused instances.
* Removes original LayerNorm modules.

**Key Implementation Details:**

* Replaces `Module._modules[module_name] = target_cls(old_module.weight, old_module.variance_epsilon).to(old_module.weight.device)` 
* Deletes replaced original module.

No notable dependencies or requirements are reported for this code snippet.


key: chroma:document
string_value:
CODE:

def _fuse_awq_mlp(model, current_module_name, fuse_module_names, module, target_cls):
    """
    Fuse the MLP layers into a target class using autoawq

    Args:
        model (`~PreTrainedModel`):
            The input pretrained model
        current_module_name (`str`):
            The current submodule name
        fuse_module_names (`List[str]`):
            The list of module names to fuse. For the MLP layers it has to be an array
            of length 3 that consists of the 3 MLP layers in the order (gate (dense layer post-attention) / up / down layers)
        module (`nn.Module`):
            The pytorch parent module that has layernorm modules to fuse
        target_cls (`~autoawq.QuantFusedMLP`):
            The `QuantFusedMLP` class as it only supports that class
            for now.
    """
    if len(fuse_module_names) == 0:
        return

    if hasattr(module, fuse_module_names[0]):
        gate_proj = getattr(module, fuse_module_names[0])
        up_proj = getattr(module, fuse_module_names[1])
        down_proj = getattr(module, fuse_module_names[2])

        previous_device = gate_proj.qweight.device

        # Deal also with the case model has `text_config` attribute
        config = model.config.get_text_config(decoder=True)
        hidden_act = config.hidden_act
        activation_fn = ACT2FN[hidden_act]
        new_module = target_cls(gate_proj, down_proj, up_proj, activation_fn)

        parent_name, child_name = current_module_name.rsplit(".", 1)
        parent = model.get_submodule(parent_name)
        setattr(parent, child_name, new_module.to(previous_device))



SUMMARY:
**Function Summary: `_fuse_awq_mlp`**

#### Purpose
Fuse the MLP layers in a PyTorch model (`model`) into a target class using autoawq (`target_cls`). Replaces the gate layer with a new `QuantFusedMLP` instance and modifies the parent module's attributes accordingly.

#### Key Implementation Details

* Calls `getattr` to access specific modules (gate_proj, up_proj, down_proj) within the input `module`.
* Sets the activation function for the new module based on the model's hidden act configuration.
* Creates a new `QuantFusedMLP` instance with the modified gate layer and preserves the parent module's device settings.
* Replaces the gate layer in the parent module with the new `QuantFusedMLP` instance using `setattr`.

#### External Dependencies

* `autoawq.QuantFusedMLP`
* `PreTrainedModel` (input model)
* `nn.Module` (parent module)

#### Notable Interfaces/Methods

* `model.config.get_text_config(decoder=True)`: returns the text configuration for the decoder.
* `ACT2FN[hidden_act]`: maps hidden activation functions to their corresponding PyTorch activation functions.


key: chroma:document
string_value:
CODE:

def _fuse_awq_attention_layers(model, module, modules_to_fuse, current_module_name, target_cls):
    """
    Fuse the Attention layers into a target class using autoawq

    Args:
        model (`~PreTrainedModel`):
            The input pretrained model
        module (`nn.Module`):
            The pytorch parent module that has layernorm modules to fuse
        modules_to_fuse (`List[str]`):
            The module fusing mapping. The dictionary has to contain a field `attention` with attention module names
            in the correct order: q, k, v, o layer
        current_module_name (`str`):
            The current submodule name
        target_cls (`~autoawq.QuantAttentionFused`):
            The `QuantAttentionFused` class as it only supports that class
            for now.
    """
    from awq.modules.linear import WQLinear_GEMM, WQLinear_GEMV

    module_has_been_fused = False

    if len(modules_to_fuse["attention"]) == 0:
        return module_has_been_fused

    if hasattr(module, modules_to_fuse["attention"][0]):
        # First, we pack the QKV layers together
        q_proj = getattr(module, modules_to_fuse["attention"][0])

        if isinstance(q_proj, WQLinear_GEMV):
            linear_target_cls = WQLinear_GEMV
            cat_dim = 0
        elif isinstance(q_proj, WQLinear_GEMM):
            linear_target_cls = WQLinear_GEMM
            cat_dim = 1
        elif is_ipex_available() and version.parse(importlib.metadata.version("autoawq")) > version.parse("0.2.6"):
            from awq.modules.linear import WQLinear_IPEX

            if isinstance(q_proj, WQLinear_IPEX):
                linear_target_cls = WQLinear_IPEX
                cat_dim = 1
        else:
            raise ValueError("Unsupported q_proj type: {type(q_proj)}")

        previous_device = q_proj.qweight.device

        k_proj = getattr(module, modules_to_fuse["attention"][1])
        v_proj = getattr(module, modules_to_fuse["attention"][2])
        o_proj = getattr(module, modules_to_fuse["attention"][3])

        bias = torch.cat([q_proj.bias, k_proj.bias, v_proj.bias], dim=0) if q_proj.bias is not None else None

        qkv_layer = linear_target_cls(
            q_proj.w_bit,
            q_proj.group_size,
            q_proj.in_features,
            q_proj.out_features + k_proj.out_features + v_proj.out_features,
            q_proj.bias is not None,
            next(iter(module.state_dict().values())).device,
        )

        qkv_layer.qweight = torch.cat([q_proj.qweight, k_proj.qweight, v_proj.qweight], dim=cat_dim)
        qkv_layer.qzeros = torch.cat([q_proj.qzeros, k_proj.qzeros, v_proj.qzeros], dim=cat_dim)
        qkv_layer.scales = torch.cat([q_proj.scales, k_proj.scales, v_proj.scales], dim=cat_dim)

        if isinstance(qkv_layer, WQLinear_GEMV):
            qkv_layer.split_k_iters = q_proj.split_k_iters

        qkv_layer.bias = bias

        fused_attention_layer = target_cls(
            modules_to_fuse["hidden_size"],
            modules_to_fuse["num_attention_heads"],
            modules_to_fuse["num_key_value_heads"],
            qkv_layer,
            o_proj,
            previous_device,
            modules_to_fuse["max_seq_len"],
            use_alibi=modules_to_fuse["use_alibi"],
            # The default value in autoawq is set to 10000.0
            rope_theta=modules_to_fuse.get("rope_theta", 10000.0),
        )

        fused_attention_layer.is_hf_transformers = True

        parent_name, child_name = current_module_name.rsplit(".", 1)
        parent = model.get_submodule(parent_name)
        setattr(parent, child_name, fused_attention_layer.to(previous_device))

        del q_proj, k_proj, v_proj, o_proj
        module_has_been_fused = True



SUMMARY:
**Function Summary: `_fuse_awq_attention_layers`**

 Fuse attention layers from a PyTorch model into the AutoAWQ QuantAttentionFused class.

### Purpose and Functionality

This function takes in a pre-trained model, its parent module with layernorm modules to fuse, modules to fuse, current submodule name, and a target class. It packs the QKV layers together and creates a new fused attention layer using autoawq's QuantAttentionFused class.

### Key Implementation Details

* Packs QKV layers from `module` into a single linear layer.
* Uses either `WQLinear_GEMM`, `WQLinear_GEMV`, or `WQLinear_IPEX` depending on the target device and version.
* Concatenates biases from various projections.
* Creates a new fused attention layer with the packed QKV layers.

### Notable Dependencies

* AutoAWQ's QuantAttentionFused class
* PyTorch parent module (`module`)
* Various linear projection classes (e.g., `WQLinear_GEMM`, `WQLinear_GEMV`, `WQLinear_IPEX`)

### Interfaces and Inherited Properties

* The fused attention layer implements `QuantAttentionFused`.
* Has attributes `qweight`, ` biases`, `qzeros`, and `scales`.


key: chroma:document
string_value:
CODE:

def post_init_awq_exllama_modules(model, exllama_config):
    """
    Runs post init for Exllama layers which performs:
        - Weights unpacking, reordering and repacking
        - Devices scratch space allocation
    """

    if exllama_config["version"] == ExllamaVersion.ONE:
        from awq.modules.linear.exllama import exllama_post_init

        model = exllama_post_init(model)
    elif exllama_config["version"] == ExllamaVersion.TWO:
        from awq.modules.linear.exllamav2 import exllamav2_post_init

        model = exllamav2_post_init(
            model,
            max_input_len=exllama_config["max_input_len"],
            max_batch_size=exllama_config["max_batch_size"],
        )
    else:
        raise ValueError(f"Unrecognized Exllama version: {exllama_config['version']}")



SUMMARY:
**Concise Summary**

`post_init_awq_exllama_modules` function:
```markdown
### Purpose and Functionality

Runs post initialization for Exllama layers, handling version-specific processing.

### Implementation Details

- Weights unpacking, reordering, and repacking (version ONE)
- Devices scratch space allocation (version TWO)

### Key Method Signatures

- `exllama_post_init(model)`
- `exllamav2_post_init(model, max_input_len, max_batch_size)`

### Dependencies and Requirements

- ExllamaVersion (enum)
- `awq.modules.linear.exllama` module
- `awq.modules.linear.exllamav2` module
```
Note: This summary preserves the technical terms, class/function names exactly, and focuses on key implementation details while excluding unnecessary information.


key: chroma:document
string_value:
CODE:

def post_init_awq_ipex_modules(model):
    """
    Runs post init for IPEX layers which performs:
        - Weights packing, reordering and repacking
    """

    from awq.modules.linear.gemm_ipex import ipex_post_init

    model = ipex_post_init(model)



SUMMARY:
**Summary:**

This function, `post_init_awq_ipex_modules`, performs post-initialization for IPEX layers in a machine learning model. It invokes the `ipex_post_init` function from `awq.modules.linear.gemm_ipex` to execute the following tasks:

* Weights packing
* Reordering
* Repacking

The updated model is returned after execution.

**Key Details:**

* Function signatures:
	+ `post_init_awq_ipex_modules(model)`
* Notable dependencies and requirements:
	+ `awq.modules.linear.gemm_ipex.ipex_post_init` (function)
* Primary purpose:
	+ Perform post-init for IPEX layers, focusing on weights manipulation.


key: chroma:document
string_value:
CODE:

def pad_to_block(tensor, dims, had_block_size, value=0):
    pad_dims = [0 for _ in range(2 * len(tensor.shape))]
    for dim in dims:
        size = tensor.shape[dim]
        next_multiple_of_1024 = ((size - 1) // had_block_size + 1) * had_block_size
        delta = next_multiple_of_1024 - size
        pad_dims[-2 * dim - 1] = delta



SUMMARY:
**Summary**

The `pad_to_block` function pads input tensor to the nearest multiple of a specified block size (`had_block_size`) in a specific set of dimensions (`dims`). It uses Kaiming's rule to calculate padding sizes and pads using PyTorch's `nn.functional.pad`.

**Key Implementation Details**

* Function takes in:
	+ `tensor`: Input tensor to be padded.
	+ `dims`: Dimensions to pad along.
	+ `had_block_size`: Block size for padding calculation.
	+ `value`: Padding value (default: 0).
* Pads using PyTorch's `nn.functional.pad` with "constant" mode and specified `value`.
* Calculates padding sizes using Kaiming's rule:
```python
next_multiple_of_1024 = ((size - 1) // had_block_size + 1) * had_block_size
delta = next_multiple_of_1024 - size
pad_dims[-2 * dim - 1] = delta
```

**Notable Dependencies/Requirements**

* PyTorch (`nn.functional` module)
* Requires specific block size calculation based on `had_block_size`.


key: chroma:document
string_value:
CODE:

def get_higgs_grid(p: int, n: int):
    if (p, n) == (2, 256):
        return torch.tensor(
            [
                [-2.501467704772949, 0.17954708635807037],
                [-0.6761789321899414, 1.2728623151779175],
                [-1.8025816679000854, 0.7613157629966736],
                [-0.538287878036499, -2.6028504371643066],
                [0.8415029644966125, -0.8600977659225464],
                [0.7023013234138489, 3.3138747215270996],
                [0.5699077844619751, 2.5782253742218018],
                [3.292393207550049, -0.6016128063201904],
                [0.5561617016792297, -1.7723814249038696],
                [-2.1012380123138428, 0.020958125591278076],
                [0.46085724234580994, 0.8428705334663391],
                [1.4548040628433228, -0.6156039237976074],
                [3.210029363632202, 0.3546904921531677],
                [0.8893890976905823, -0.5967988967895508],
                [0.8618854284286499, -3.2061192989349365],
                [1.1360996961593628, -0.23852407932281494],
                [1.6646337509155273, -0.9265465140342712],
                [1.4767773151397705, 1.2476022243499756],
                [-1.0511897802352905, 1.94503915309906],
                [-1.56318998336792, -0.3264186680316925],
                [-0.1829211413860321, 0.2922491431236267],
                [-0.8950616717338562, -1.3887052536010742],
                [-0.08206957578659058, -1.329533576965332],
                [-0.487422913312912, 1.4817842245101929],
                [-1.6769757270812988, -2.8269758224487305],
                [-1.5057679414749146, 1.8905963897705078],
                [1.8335362672805786, 1.0515104532241821],
                [0.3273945450782776, 1.0491033792495728],
                [-3.295924186706543, -0.7021600008010864],
                [-1.8428784608840942, -1.2315762042999268],
                [-0.8575026392936707, -1.7005949020385742],
                [-1.120667815208435, 0.6467998027801514],
                [-0.1588846743106842, -1.804071068763733],
                [-0.8539647459983826, 0.5645008683204651],
                [-1.4192019701004028, -0.6175029873847961],
                [1.0799058675765991, 1.7871345281600952],
                [1.171311855316162, 0.7511613965034485],
                [2.162078380584717, 0.8044339418411255],
                [1.3969420194625854, -1.243762493133545],
                [-0.23818807303905487, 0.053944624960422516],
                [2.304199457168579, -1.2667627334594727],
                [1.4225027561187744, 0.568610668182373],
                [0.376836895942688, -0.7134661674499512],
                [2.0404467582702637, 0.4087389409542084],
                [0.7639489769935608, -1.1367933750152588],
                [0.3622530400753021, -1.4827953577041626],
                [0.4100743532180786, 0.36108437180519104],
                [-1.5867475271224976, -1.618212342262268],
                [-2.2769672870635986, -1.2132309675216675],
                [0.9184022545814514, -0.34428009390830994],
                [-0.3902314603328705, 0.21785245835781097],
                [3.120687484741211, 1.3077973127365112],
                [1.587440848350525, -1.6506884098052979],
                [-1.718808889389038, -0.038405973464250565],
                [-0.6888407468795776, -0.8402308821678162],
                [-0.7981445789337158, -1.1117373704910278],
                [-2.4124443531036377, 1.3419722318649292],
                [-0.6611530184745789, 0.9939885139465332],
                [-0.33103418350219727, -0.16702833771705627],
                [-2.4091389179229736, -2.326857566833496],
                [1.6610108613967896, -2.159703254699707],
                [0.014884627424180508, 0.3887578248977661],
                [0.029668325558304787, 1.8786455392837524],
                [1.180362582206726, 2.699317216873169],
                [1.821286678314209, -0.5960053205490112],
                [-0.44835323095321655, 3.327436685562134],
                [-0.3714401423931122, -2.1466753482818604],
                [-1.1103475093841553, -2.4536871910095215],
                [-0.39110705256462097, 0.6670510172843933],
                [0.474752813577652, -1.1959707736968994],
                [-0.013110585510730743, -2.52519154548645],
                [-2.0836575031280518, -1.703289270401001],
                [-1.1077687740325928, -0.1252644956111908],
                [-0.4138077199459076, 1.1837692260742188],
                [-1.977599024772644, 1.688241720199585],
                [-1.659559965133667, -2.1387736797332764],
                [0.03242531046271324, 0.6526556015014648],
                [0.9127950072288513, 0.6099498867988586],
                [-0.38478314876556396, 0.433487206697464],
                [0.27454206347465515, -0.27719801664352417],
                [0.10388526320457458, 2.2812814712524414],
                [-0.014394169673323631, -3.177137613296509],
                [-1.2871228456497192, -0.8961855173110962],
                [0.5720916986465454, -0.921597957611084],
                [1.1159656047821045, -0.7609877586364746],
                [2.4383342266082764, -2.2983546257019043],
                [-0.294057160615921, -0.9770799875259399],
                [-0.9342701435089111, 1.107579231262207],
                [-1.549338698387146, 3.090520143508911],
                [2.6076579093933105, 2.051239013671875],
                [-0.9259037375450134, 1.407211184501648],
                [-0.1747353971004486, 0.540488600730896],
                [-0.8963701725006104, 0.8271111249923706],
                [0.6480194926261902, 1.0128909349441528],
                [0.980783998966217, -0.06156221032142639],
                [-0.16883476078510284, 1.0601658821105957],
                [0.5839992761611938, 0.004697148688137531],
                [-0.34228450059890747, -1.2423977851867676],
                [2.500824451446533, 0.3665279746055603],
                [-0.17641609907150269, 1.3529551029205322],
                [0.05378641560673714, 2.817232847213745],
                [-1.2391047477722168, 2.354328155517578],
                [0.630434513092041, -0.668536365032196],
                [1.7576488256454468, 0.6738647818565369],
                [0.4435231387615204, 0.6000469326972961],
                [-0.08794835954904556, -0.11511358618736267],
                [1.6540337800979614, 0.33995017409324646],
                [-0.04202975332736969, -0.5375117063522339],
                [-0.4247745871543884, -0.7897617220878601],
                [0.06695003807544708, 1.2000739574432373],
                [-3.2508881092071533, 0.28734830021858215],
                [-1.613816261291504, 0.4944162368774414],
                [1.3598989248275757, 0.26117825508117676],
                [2.308382511138916, 1.3462618589401245],
                [-1.2137469053268433, -1.9254342317581177],
                [-0.4889402985572815, 1.8136259317398071],
                [-0.1870335340499878, -0.3480615019798279],
                [1.0766386985778809, -1.0627082586288452],
                [0.4651014506816864, 2.131748914718628],
                [-0.1306295394897461, -0.7811847925186157],
                [0.06433182954788208, -1.5397958755493164],
                [-0.2894323468208313, -0.5789554715156555],
                [-0.6081662178039551, 0.4845278263092041],
                [2.697964668273926, -0.18515698611736298],
                [0.1277363896369934, -0.7221432328224182],
                [0.8700758218765259, 0.35042452812194824],
                [0.22088994085788727, 0.495242178440094],
                [-2.5843818187713623, -0.8000828623771667],
                [0.6732649803161621, -1.4362232685089111],
                [-1.5286413431167603, 1.0417330265045166],
                [-1.1222513914108276, -0.6269875764846802],
                [-0.9752035140991211, -0.8750635385513306],
                [-2.6369473934173584, 0.6918523907661438],
                [0.14478731155395508, -0.041986867785453796],
                [-1.5629483461380005, 1.4369450807571411],
                [0.38952457904815674, -2.16428804397583],
                [-0.16885095834732056, 0.7976621985435486],
                [-3.12416934967041, 1.256506085395813],
                [0.6843105554580688, -0.4203019142150879],
                [1.9345275163650513, 1.934950351715088],
                [0.012184220366179943, -2.1080918312072754],
                [-0.6350273489952087, 0.7358828186988831],
                [-0.837304949760437, -0.6214472651481628],
                [0.08211923390626907, -0.9472538232803345],
                [2.9332995414733887, -1.4956780672073364],
                [1.3806978464126587, -0.2916182279586792],
                [0.06773144006729126, 0.9285762310028076],
                [-1.1943119764328003, 1.5963770151138306],
                [1.6395620107650757, -0.32285431027412415],
                [-1.390851378440857, -0.08273141086101532],
                [1.816330909729004, -1.2812227010726929],
                [0.7921574711799622, -2.1135804653167725],
                [0.5817914605140686, 1.2644577026367188],
                [1.929347038269043, -0.2386285960674286],
                [0.8877345323562622, 1.190008521080017],
                [1.4732073545455933, 0.8935023546218872],
                [-2.8518524169921875, -1.5478795766830444],
                [0.2439267635345459, 0.7576767802238464],
                [0.5246709585189819, -2.606659412384033],
                [1.150876760482788, 1.4073830842971802],
                [-0.2643202245235443, 2.0634236335754395],
                [1.555483341217041, -0.0023102816194295883],
                [2.0830578804016113, -1.7225427627563477],
                [-0.5424830317497253, -1.070199728012085],
                [0.9168899655342102, 0.8955540060997009],
                [-0.8120972514152527, 2.696739912033081],
                [-0.29908373951911926, -1.5310651063919067],
                [1.2320337295532227, -1.556247353553772],
                [1.8612544536590576, 0.08704725652933121],
                [0.22133447229862213, -1.8091708421707153],
                [-0.4403655230998993, -0.38571012020111084],
                [-1.88539457321167, 1.192205786705017],
                [2.239687919616699, 0.004709010478109121],
                [1.139495611190796, 0.45733731985092163],
                [-1.507995367050171, 0.19716016948223114],
                [0.46986445784568787, 1.5422041416168213],
                [-1.2573751211166382, -0.35984551906585693],
                [-1.7415345907211304, -0.6020717024803162],
                [1.0751984119415283, 0.19006384909152985],
                [2.24186635017395, -0.46343153715133667],
                [0.3610347509384155, -0.07658443599939346],
                [-1.3111497163772583, 0.432013601064682],
                [0.6164408326148987, 0.24538464844226837],
                [-1.9266542196273804, -0.3256155550479889],
                [-0.5870336890220642, -0.1879584938287735],
                [-1.0476511716842651, 0.3677721917629242],
                [-1.229940414428711, 1.2433830499649048],
                [0.18550436198711395, 0.22753673791885376],
                [-0.017921989783644676, 0.12625974416732788],
                [1.1659504175186157, -0.5020995736122131],
                [-0.5983408093452454, -1.40438973903656],
                [0.7519024014472961, -0.16282692551612854],
                [0.9920787811279297, -1.344896912574768],
                [-0.8103678226470947, 0.3064485788345337],
                [0.6956969499588013, 1.8208192586898804],
                [-2.7830491065979004, -0.2299390584230423],
                [-0.34681546688079834, 2.4890666007995605],
                [-1.4452646970748901, -1.2216600179672241],
                [-2.1872897148132324, 0.8926076292991638],
                [1.706072211265564, -2.8440372943878174],
                [1.1119003295898438, -2.4923460483551025],
                [-2.582794666290283, 2.0973289012908936],
                [0.04987720400094986, -0.2964983284473419],
                [-2.063807487487793, -0.7847916483879089],
                [-0.4068813621997833, 0.9135897755622864],
                [-0.9814359545707703, -0.3874954879283905],
                [-1.4227229356765747, 0.7337291240692139],
                [0.3065044581890106, 1.3125417232513428],
                [1.2160996198654175, -1.9643305540084839],
                [-1.2163853645324707, 0.14608727395534515],
                [-2.3030710220336914, -0.37558120489120483],
                [0.9232977628707886, 2.1843791007995605],
                [-0.1989777386188507, 1.651851773262024],
                [-0.714374840259552, -0.39365994930267334],
                [-0.7805715799331665, -2.099881887435913],
                [0.9015759229660034, -1.7053706645965576],
                [0.1033422127366066, 1.5256654024124146],
                [-1.8773194551467896, 2.324174165725708],
                [1.9227174520492554, 2.7441604137420654],
                [-0.5994020104408264, 0.23984014987945557],
                [1.3496100902557373, -0.9126054644584656],
                [-0.8765304088592529, -3.1877026557922363],
                [-1.2040035724639893, -1.5169521570205688],
                [1.4261796474456787, 2.150200128555298],
                [1.463774561882019, 1.6656692028045654],
                [0.20364105701446533, -0.4988172650337219],
                [0.5195154547691345, -0.24067887663841248],
                [-1.1116786003112793, -1.1599653959274292],
                [-0.8490808606147766, -0.1681060940027237],
                [0.3189965784549713, -0.9641751646995544],
                [-0.5664751529693604, -0.5951744318008423],
                [-1.6347930431365967, -0.9137664437294006],
                [0.44048091769218445, -0.47259435057640076],
                [-2.147747039794922, 0.47442489862442017],
                [1.834734320640564, 1.4462147951126099],
                [1.1777573823928833, 1.0659226179122925],
                [-0.9568989872932434, 0.09495053440332413],
                [-1.838529348373413, 0.2950586676597595],
                [-0.4800611734390259, 0.014894310384988785],
                [-0.5235516428947449, -1.7687653303146362],
                [2.0735011100769043, -0.8825281262397766],
                [2.637502431869507, 0.8455678224563599],
                [2.606602907180786, -0.7848446369171143],
                [-1.1886937618255615, 0.9330510497093201],
                [0.38082656264305115, 0.13328030705451965],
                [0.6847941875457764, 0.7384101152420044],
                [1.2638574838638306, -0.007309418171644211],
                [0.18292222917079926, -1.22371244430542],
                [0.8143821954727173, 1.4976691007614136],
                [0.6571850776672363, 0.48368802666664124],
                [-0.6991601586341858, 2.150190830230713],
                [0.8101756572723389, 0.10206498205661774],
                [-0.08768226951360703, -1.084917664527893],
                [-0.7208092212677002, 0.03657956421375275],
                [0.3211449086666107, 1.803687334060669],
                [-0.7835946083068848, 1.6869111061096191],
            ]
        )
    if (p, n) == (2, 64):
        return torch.tensor(
            [
                [-2.7216711044311523, 0.14431366324424744],
                [-0.766914427280426, 1.7193410396575928],
                [-2.2575762271881104, 1.2476624250411987],
                [1.233758807182312, -2.3560616970062256],
                [0.8701965808868408, -0.2649352252483368],
                [1.4506438970565796, 2.1776366233825684],
                [-0.06305818259716034, 1.9049758911132812],
                [2.536226511001587, 0.563927412033081],
                [0.4599496126174927, -1.8745561838150024],
                [-1.900517225265503, -0.30703988671302795],
                [0.09386251866817474, 0.8755807280540466],
                [1.946500539779663, -0.6743080615997314],
                [2.1338934898376465, 1.4581491947174072],
                [0.9429940581321716, -0.8038390278816223],
                [2.0697755813598633, -1.614896535873413],
                [0.772676408290863, 0.22017823159694672],
                [1.0689979791641235, -1.525044322013855],
                [0.6813604831695557, 1.1345642805099487],
                [0.4706456661224365, 2.606626272201538],
                [-1.294018030166626, -0.4372096061706543],
                [-0.09134224057197571, 0.4610418677330017],
                [-0.7907772064208984, -0.48412787914276123],
                [0.060459110885858536, -0.9172890186309814],
                [-0.5855047702789307, 2.56172513961792],
                [0.11484206467866898, -2.659848213195801],
                [-1.5893300771713257, 2.188580274581909],
                [1.6750942468643188, 0.7089915871620178],
                [-0.445697546005249, 0.7452405095100403],
                [-1.8539940118789673, -1.8377939462661743],
                [-1.5791912078857422, -1.017285943031311],
                [-1.030419945716858, -1.5746369361877441],
                [-1.9511750936508179, 0.43696075677871704],
                [-0.3446580767631531, -1.8953213691711426],
                [-1.4219647645950317, 0.7676230669021606],
                [-0.9191089272499084, 0.5021472573280334],
                [0.20464491844177246, 1.3684605360031128],
                [0.5402919054031372, 0.6699410676956177],
                [1.8903915882110596, 0.03638288006186485],
                [0.4723062515258789, -0.6216739416122437],
                [-0.41345009207725525, -0.22752176225185394],
                [2.7119064331054688, -0.5111885070800781],
                [1.065286636352539, 0.6950305700302124],
                [0.40629103779792786, -0.14339995384216309],
                [1.2815024852752686, 0.17108257114887238],
                [0.01785222627222538, -0.43778058886528015],
                [0.054590027779340744, -1.4225547313690186],
                [0.3076786696910858, 0.30697619915008545],
                [-0.9498570561408997, -0.9576997756958008],
                [-2.4640724658966064, -0.9660449028015137],
                [1.3714425563812256, -0.39760473370552063],
                [-0.4857747256755829, 0.2386789172887802],
                [1.2797833681106567, 1.3097363710403442],
                [0.5508887767791748, -1.1777795553207397],
                [-1.384316325187683, 0.1465839296579361],
                [-0.46556955575942993, -1.2442727088928223],
                [-0.3915477693080902, -0.7319604158401489],
                [-1.4005504846572876, 1.3890998363494873],
                [-0.8647305965423584, 1.0617644786834717],
                [-0.8901953101158142, -0.01650036871433258],
                [-0.9893633723258972, -2.4662880897521973],
                [1.445534110069275, -1.049334168434143],
                [-0.041650623083114624, 0.012734669260680676],
                [-0.3302375078201294, 1.26217782497406],
                [0.6934980154037476, 1.7714335918426514],
            ]
        )
    elif (p, n) == (2, 16):
        return torch.tensor(
            [
                [-0.8996632695198059, -1.6360418796539307],
                [-0.961183488368988, 1.5999565124511719],
                [-1.882026195526123, 0.678778350353241],
                [0.36300793290138245, -1.9667866230010986],
                [-0.6814072728157043, -0.576818585395813],
                [0.7270012497901917, 0.6186859607696533],
                [0.3359416127204895, 1.8371193408966064],
                [1.859930396080017, 0.036668598651885986],
                [0.17208248376846313, -0.9401724338531494],
                [-1.7599700689315796, -0.6244229674339294],
                [-0.8993809223175049, 0.32267823815345764],
                [0.839488685131073, -0.3017036020755768],
                [1.5314953327178955, 1.2942044734954834],
                [-0.0011779458727687597, 0.00022069070837460458],
                [1.4274526834487915, -1.207889199256897],
                [-0.16123905777931213, 0.8787511587142944],
            ]
        )
    elif (p, n) == (1, 16):
        return torch.tensor(
            [
                [-2.7325894832611084],
                [-2.069017171859741],
                [-1.6180464029312134],
                [-1.2562311887741089],
                [-0.9423404335975647],
                [-0.6567591428756714],
                [-0.38804829120635986],
                [-0.12839503586292267],
                [0.12839503586292267],
                [0.38804829120635986],
                [0.6567591428756714],
                [0.9423404335975647],
                [1.2562311887741089],
                [1.6180464029312134],
                [2.069017171859741],
                [2.7325894832611084],
            ]
        )
    elif (p, n) == (1, 8):
        return torch.tensor(
            [
                [-2.1519455909729004],
                [-1.3439092636108398],
                [-0.7560052871704102],
                [-0.2450941801071167],
                [0.2450941801071167],
                [0.7560052871704102],
                [1.3439092636108398],
                [2.1519455909729004],
            ]
        )
    elif (p, n) == (1, 4):
        return torch.tensor([[-1.5104175806045532], [-0.4527800381183624], [0.4527800381183624], [1.5104175806045532]])
    else:


SUMMARY:
The code defines a function `torch.tensor()` that generates 1D or 2D tensors with specific shapes and values, possibly for testing or demonstration purposes.

Here's a concise summary:

- The function takes two arguments: `p` (the power of the tensor) and `n` (the length/middle dimension of the tensor).
- The functions handle three different cases:
    - `(1, n)`: For 1D tensors with `n` elements.
    - `(2, n)`: For 2D tensors with `n` rows/columns.
    - `(3, n) and (4, n)` are currently commented out and may be for future extension or other test cases.

The `p=2` case, mentioned in the explanation but not actually used, seems to refer to the power of a tensor similar to powers of 1/√2 (used in some linear algebra operations like matrix multiplication).


