# embedding_metadata

key: chroma:document
string_value:
CODE:

def _is_end_of_word(text):
    """Checks whether the last character in text is one of a punctuation, control or whitespace character."""
    last_char = text[-1]


SUMMARY:
**Summary: `bool _is_end_of_word(text)`**

**Purpose:** Determines if the input `text` ends with punctuation, control, or whitespace character.

**Behavior:**

* Examines the last character of `text`.
* Uses bitwise OR (`|`) to check if it's a control (`_is_control()`), punctuation (`_is_punctuation()`), or whitespace (`_is_whitespace()`) character.
* Returns `True` if any condition is met, indicating the end of word.

**Key Functions:** `_is_control(last_char)`, `_is_punctuation(last_char)`, `_is_whitespace(last_char)`


key: chroma:document
string_value:
CODE:

def _is_start_of_word(text):
    """Checks whether the first character in text is one of a punctuation, control or whitespace character."""
    first_char = text[0]


SUMMARY:
**Summary:**

* **Functionality:** Determines whether the first character of a given `text` is a punctuation, control, or whitespace character.
* **Key Implementation Details:**
	+ Checks first character of text using `_is_control(first_char)`, `_is_punctuation(first_char)`, and `_is_whitespace(first_char)` helper functions.
	+ Uses bit-wise OR operator (`|`) to combine the results of individual checks.
* **Notable Dependencies/Requirements:**
	+ None explicit dependencies listed, but assumes existence of `_is_control` and `_is_punctuation` functions with similar signature.

This summary aims to provide a concise overview of the function's purpose, implementation, and context-dependent details while preserving precision in technical terms.


key: chroma:document
string_value:
CODE:

def _insert_one_token_to_ordered_list(token_list: List[str], new_token: str):
    """
    Inserts one token to an ordered list if it does not already exist. Note: token_list must be sorted.
    """
    insertion_idx = bisect.bisect_left(token_list, new_token)
    # Checks if new_token is already in the ordered token_list
    if insertion_idx < len(token_list) and token_list[insertion_idx] == new_token:
        # new_token is in token_list, don't add
        return
    else:


SUMMARY:
Here's a concise summary of the function:

**Function Summary:**
 `_insert_one_token_to_ordered_list(token_list: List[str], new_token: str)`

**Main Purpose and Functionality:**

This function inserts a new token into an already sorted ordered list if it does not already exist. It ensures uniqueness by checking the sorted list with `bisect.bisect_left()`.

**Key Implementation Details:**

* Inserts token at correct insertion index using `token_list.insert()`
* Uses `bisect.bisect_left()` to efficiently find insertion point

**Notable Dependencies and Requirements:**

* Python's built-in `bisect` module for ordered search
* `List[str]` type hint indicating the expected input list is a sorted list of strings.


key: chroma:document
string_value:
CODE:

def to_channel_dimension_format(
    image: np.ndarray,
    channel_dim: Union[ChannelDimension, str],
    input_channel_dim: Optional[Union[ChannelDimension, str]] = None,
) -> np.ndarray:
    """
    Converts `image` to the channel dimension format specified by `channel_dim`.

    Args:
        image (`numpy.ndarray`):
            The image to have its channel dimension set.
        channel_dim (`ChannelDimension`):
            The channel dimension format to use.
        input_channel_dim (`ChannelDimension`, *optional*):
            The channel dimension format of the input image. If not provided, it will be inferred from the input image.

    Returns:
        `np.ndarray`: The image with the channel dimension set to `channel_dim`.
    """
    if not isinstance(image, np.ndarray):
        raise TypeError(f"Input image must be of type np.ndarray, got {type(image)}")

    if input_channel_dim is None:
        input_channel_dim = infer_channel_dimension_format(image)

    target_channel_dim = ChannelDimension(channel_dim)
    if input_channel_dim == target_channel_dim:
        return image

    if target_channel_dim == ChannelDimension.FIRST:
        image = image.transpose((2, 0, 1))
    elif target_channel_dim == ChannelDimension.LAST:
        image = image.transpose((1, 2, 0))
    else:
        raise ValueError("Unsupported channel dimension format: {}".format(channel_dim))



SUMMARY:
**Summary**

`to_channel_dimension_format` function converts an `image` numpy array to the specified `channel_dim`. It takes into account the input's inferred channel dimension if not provided.

*   Ensures the input is a numpy.ndarray and raises a TypeError for non-numpy values.
*   Inferences or uses the input's `ChannelDimension` format if not provided.
*   Modifies the image based on the target channel dimension, swapping dimensions as necessary.
*   Returns the modified image with its channel dimension set.

**Key Implementation Details**

*   Uses numpy array transposition methods for converting channel dimension.
*   Provides type checking and input validation.

**Notable Dependencies and Requirements**

*   `numpy` library (imported but not shown).
*   `ChannelDimension` enum or class (assumed, references are not provided).


key: chroma:document
string_value:
CODE:

def rescale(
    image: np.ndarray,
    scale: float,
    data_format: Optional[ChannelDimension] = None,
    dtype: np.dtype = np.float32,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray:
    """
    Rescales `image` by `scale`.

    Args:
        image (`np.ndarray`):
            The image to rescale.
        scale (`float`):
            The scale to use for rescaling the image.
        data_format (`ChannelDimension`, *optional*):
            The channel dimension format of the image. If not provided, it will be the same as the input image.
        dtype (`np.dtype`, *optional*, defaults to `np.float32`):
            The dtype of the output image. Defaults to `np.float32`. Used for backwards compatibility with feature
            extractors.
        input_data_format (`ChannelDimension`, *optional*):
            The channel dimension format of the input image. If not provided, it will be inferred from the input image.

    Returns:
        `np.ndarray`: The rescaled image.
    """
    if not isinstance(image, np.ndarray):
        raise TypeError(f"Input image must be of type np.ndarray, got {type(image)}")

    rescaled_image = image.astype(np.float64) * scale  # Numpy type promotion has changed, so always upcast first
    if data_format is not None:
        rescaled_image = to_channel_dimension_format(rescaled_image, data_format, input_data_format)

    rescaled_image = rescaled_image.astype(dtype)  # Finally downcast to the desired dtype at the end



SUMMARY:
**Rescaling Function Summary**

### Purpose and Functionality
Rescales an input image (`image`) by a provided scale factor.

### Implementation Details

*   **Scaling**: The function first casts the input image to float64 and multiplies it by the scale factor.
*   **Channel Dimension Format**: If `data_format` is specified, the rescaled image is converted back to this format using the `to_channel_dimension_format` function. Otherwise, the original channel dimension format is preserved.
*   **Data Type Conversion**: The final step is casting the rescaled image to the desired data type (`dtype`).

### Notable Dependencies and Requirements

*   Numpy arrays (`np.ndarray`)
*   Channel dimension format classes (`ChannelDimension`, `Optional[ChannelDimension]`)
*   `to_channel_dimension_format` function
*   Optional inputs:

    *   `data_format`: specified channel dimension format (default=None)

### API Signature

```python
def rescale(
    image: np.ndarray,
    scale: float,
    data_format: Optional[ChannelDimension] = None, 
    dtype: np.dtype = np.float32,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray:
```

### Return Value

The rescaled image as a NumPy array.

```python
np.ndarray: The rescaled image.
```


key: chroma:document
string_value:
CODE:

def _rescale_for_pil_conversion(image):
    """
    Detects whether or not the image needs to be rescaled before being converted to a PIL image.

    The assumption is that if the image is of type `np.float` and all values are between 0 and 1, it needs to be
    rescaled.
    """
    if image.dtype == np.uint8:
        do_rescale = False
    elif np.allclose(image, image.astype(int)):
        if np.all(0 <= image) and np.all(image <= 255):
            do_rescale = False
        else:
            raise ValueError(
                "The image to be converted to a PIL image contains values outside the range [0, 255], "
                f"got [{image.min()}, {image.max()}] which cannot be converted to uint8."
            )
    elif np.all(0 <= image) and np.all(image <= 1):
        do_rescale = True
    else:
        raise ValueError(
            "The image to be converted to a PIL image contains values outside the range [0, 1], "
            f"got [{image.min()}, {image.max()}] which cannot be converted to uint8."
        )


SUMMARY:
**Summary**

The `_rescale_for_pil_conversion` function determines whether an image needs rescaling before conversion to a PIL image. It checks the image's data type and values:

- If the image is of type `npuint8`, no rescaling is needed.
- If all values are between 0 and UV255, no rescaling is required but must be within this range.
- If all values are between 0 and 1, and not outside it, rescaling is recommended.

If the image contains values outside the acceptable ranges, a `ValueError` is raised.

**Key Implementation Details**

* The function uses `np.allclose`, `np.all`, and comparisons to determine whether the image requires rescaling.
* It raises a `ValueError` when the image's values are outside specified ranges.
* The function takes no other inputs except the `image` parameter.


key: chroma:document
string_value:
CODE:

def to_pil_image(
    image: Union[np.ndarray, "PIL.Image.Image", "torch.Tensor", "tf.Tensor", "jnp.ndarray"],
    do_rescale: Optional[bool] = None,
    image_mode: Optional[str] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> "PIL.Image.Image":
    """
    Converts `image` to a PIL Image. Optionally rescales it and puts the channel dimension back as the last axis if
    needed.

    Args:
        image (`PIL.Image.Image` or `numpy.ndarray` or `torch.Tensor` or `tf.Tensor`):
            The image to convert to the `PIL.Image` format.
        do_rescale (`bool`, *optional*):
            Whether or not to apply the scaling factor (to make pixel values integers between 0 and 255). Will default
            to `True` if the image type is a floating type and casting to `int` would result in a loss of precision,
            and `False` otherwise.
        image_mode (`str`, *optional*):
            The mode to use for the PIL image. If unset, will use the default mode for the input image type.
        input_data_format (`ChannelDimension`, *optional*):
            The channel dimension format of the input image. If unset, will use the inferred format from the input.

    Returns:
        `PIL.Image.Image`: The converted image.
    """
    requires_backends(to_pil_image, ["vision"])

    if isinstance(image, PIL.Image.Image):
        return image

    # Convert all tensors to numpy arrays before converting to PIL image
    if is_torch_tensor(image) or is_tf_tensor(image):
        image = image.numpy()
    elif is_jax_tensor(image):
        image = np.array(image)
    elif not isinstance(image, np.ndarray):
        raise ValueError("Input image type not supported: {}".format(type(image)))

    # If the channel has been moved to first dim, we put it back at the end.
    image = to_channel_dimension_format(image, ChannelDimension.LAST, input_data_format)

    # If there is a single channel, we squeeze it, as otherwise PIL can't handle it.
    image = np.squeeze(image, axis=-1) if image.shape[-1] == 1 else image

    # PIL.Image can only store uint8 values so we rescale the image to be between 0 and 255 if needed.
    do_rescale = _rescale_for_pil_conversion(image) if do_rescale is None else do_rescale

    if do_rescale:
        image = rescale(image, 255)

    image = image.astype(np.uint8)


SUMMARY:
**Function Summary**

`to_pil_image`: Converts input image to PIL Image format, optionally resizing and normalizing channel dimensions.

**Primary Purpose**
Converts diverse image formats (numpy, torch, tf, jnp) to PIL Image while preserving accuracy and adhering to specifications.

**Implementation Details**

1.  **Input Validation**: Checks if input is a supported type, converts tensor representations to numpy arrays.
2.  **Channel Dimension Reformatting**: Restores channel dimension format to last axis if required.
3.  **Single Channel Squeezing**: Removes singleton channels from image data for PIL compatibility.
4.  **Optional Rescaling and Normalization**: Applies rescaling factor depending on input type, ensuring pixel values fit within [0, 255] range.
5.  **Data Type Conversion**: Converts output to uint8 format using `np.uint8`.
6.  **PIL Image Creation**: Returns a PIL Image representation of the final processed image.

**Dependencies and Requirements**

*   `torch`, `tf`, `jax`
*   External PIL, numpy
*   Uses `_rescale_for_pil_conversion` function from private module

Note: The original code is preserved unchanged in this summary.


key: docstring
string_value:
# Logic adapted from torchvision resizing logic: https://github.com/pytorch/vision/blob/511924c1ced4ce0461197e5caa64ce5b9e558aab/torchvision/transforms/functional.py#L366


key: chroma:document
string_value:
CODE:
# Logic adapted from torchvision resizing logic: https://github.com/pytorch/vision/blob/511924c1ced4ce0461197e5caa64ce5b9e558aab/torchvision/transforms/functional.py#L366
def get_resize_output_image_size(
    input_image: np.ndarray,
    size: Union[int, Tuple[int, int], List[int], Tuple[int]],
    default_to_square: bool = True,
    max_size: Optional[int] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> tuple:
    """
    Find the target (height, width) dimension of the output image after resizing given the input image and the desired
    size.

    Args:
        input_image (`np.ndarray`):
            The image to resize.
        size (`int` or `Tuple[int, int]` or List[int] or `Tuple[int]`):
            The size to use for resizing the image. If `size` is a sequence like (h, w), output size will be matched to
            this.

            If `size` is an int and `default_to_square` is `True`, then image will be resized to (size, size). If
            `size` is an int and `default_to_square` is `False`, then smaller edge of the image will be matched to this
            number. i.e, if height > width, then image will be rescaled to (size * height / width, size).
        default_to_square (`bool`, *optional*, defaults to `True`):
            How to convert `size` when it is a single int. If set to `True`, the `size` will be converted to a square
            (`size`,`size`). If set to `False`, will replicate
            [`torchvision.transforms.Resize`](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Resize)
            with support for resizing only the smallest edge and providing an optional `max_size`.
        max_size (`int`, *optional*):
            The maximum allowed for the longer edge of the resized image: if the longer edge of the image is greater
            than `max_size` after being resized according to `size`, then the image is resized again so that the longer
            edge is equal to `max_size`. As a result, `size` might be overruled, i.e the smaller edge may be shorter
            than `size`. Only used if `default_to_square` is `False`.
        input_data_format (`ChannelDimension`, *optional*):
            The channel dimension format of the input image. If unset, will use the inferred format from the input.

    Returns:
        `tuple`: The target (height, width) dimension of the output image after resizing.
    """
    if isinstance(size, (tuple, list)):
        if len(size) == 2:
            return tuple(size)
        elif len(size) == 1:
            # Perform same logic as if size was an int
            size = size[0]
        else:
            raise ValueError("size must have 1 or 2 elements if it is a list or tuple")

    if default_to_square:
        return (size, size)

    height, width = get_image_size(input_image, input_data_format)
    short, long = (width, height) if width <= height else (height, width)
    requested_new_short = size

    new_short, new_long = requested_new_short, int(requested_new_short * long / short)

    if max_size is not None:
        if max_size <= requested_new_short:
            raise ValueError(
                f"max_size = {max_size} must be strictly greater than the requested "
                f"size for the smaller edge size = {size}"
            )
        if new_long > max_size:
            new_short, new_long = int(max_size * new_short / new_long), max_size



SUMMARY:
**Concise Summary**

This function, `get_resize_output_image_size`, calculates the target size of an output image after resizing given the input image and desired size. It takes into account the options to default to square cropping or replicate a transform like `torchvision.transforms.Resize`.

**Key Implementation Details**

*   The function uses a tuple to represent the input size, allowing for flexibility in specifying dimensions (e.g., `(height, width)`, `[width]`, `height`, or `int`).
*   When converting a single integer value to a tuple, it defaults to squaring the value if specified should default to square cropping (`default_to_square=True`).
*   The function first determines whether the shorter or longer edge of the input image should be used to calculate the output size based on the specified "size" value.
*   It also enforces maximum allowed dimensions for the longer edge of the resized image using the `max_size` parameter.

**Return Value**

The function returns a tuple representing the target (height, width) dimension of the output image after resizing.


key: chroma:document
string_value:
CODE:

def resize(
    image: np.ndarray,
    size: Tuple[int, int],
    resample: "PILImageResampling" = None,
    reducing_gap: Optional[int] = None,
    data_format: Optional[ChannelDimension] = None,
    return_numpy: bool = True,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray:
    """
    Resizes `image` to `(height, width)` specified by `size` using the PIL library.

    Args:
        image (`np.ndarray`):
            The image to resize.
        size (`Tuple[int, int]`):
            The size to use for resizing the image.
        resample (`int`, *optional*, defaults to `PILImageResampling.BILINEAR`):
            The filter to user for resampling.
        reducing_gap (`int`, *optional*):
            Apply optimization by resizing the image in two steps. The bigger `reducing_gap`, the closer the result to
            the fair resampling. See corresponding Pillow documentation for more details.
        data_format (`ChannelDimension`, *optional*):
            The channel dimension format of the output image. If unset, will use the inferred format from the input.
        return_numpy (`bool`, *optional*, defaults to `True`):
            Whether or not to return the resized image as a numpy array. If False a `PIL.Image.Image` object is
            returned.
        input_data_format (`ChannelDimension`, *optional*):
            The channel dimension format of the input image. If unset, will use the inferred format from the input.

    Returns:
        `np.ndarray`: The resized image.
    """
    requires_backends(resize, ["vision"])

    resample = resample if resample is not None else PILImageResampling.BILINEAR

    if not len(size) == 2:
        raise ValueError("size must have 2 elements")

    # For all transformations, we want to keep the same data format as the input image unless otherwise specified.
    # The resized image from PIL will always have channels last, so find the input format first.
    if input_data_format is None:
        input_data_format = infer_channel_dimension_format(image)
    data_format = input_data_format if data_format is None else data_format

    # To maintain backwards compatibility with the resizing done in previous image feature extractors, we use
    # the pillow library to resize the image and then convert back to numpy
    do_rescale = False
    if not isinstance(image, PIL.Image.Image):
        do_rescale = _rescale_for_pil_conversion(image)
        image = to_pil_image(image, do_rescale=do_rescale, input_data_format=input_data_format)
    height, width = size
    # PIL images are in the format (width, height)
    resized_image = image.resize((width, height), resample=resample, reducing_gap=reducing_gap)

    if return_numpy:
        resized_image = np.array(resized_image)
        # If the input image channel dimension was of size 1, then it is dropped when converting to a PIL image
        # so we need to add it back if necessary.
        resized_image = np.expand_dims(resized_image, axis=-1) if resized_image.ndim == 2 else resized_image
        # The image is always in channels last format after converting from a PIL image
        resized_image = to_channel_dimension_format(
            resized_image, data_format, input_channel_dim=ChannelDimension.LAST
        )
        # If an image was rescaled to be in the range [0, 255] before converting to a PIL image, then we need to
        # rescale it back to the original range.
        resized_image = rescale(resized_image, 1 / 255) if do_rescale else resized_image


SUMMARY:
**Function Summary: `resize`**

Resizes an image to specified size using PIL library.

### Purpose and Functionality
The primary purpose of this function is to resize an input image (`image`) to a desired size ( `size` ) while applying various resampling filters, optimization techniques, and channel dimension format adjustments. The function takes several optional parameters to customize the resizing process.

### Key Implementation Details

*   Usage of PIL library for resizing images
*   Application of resampling filters (default: Bilinear) via `resample` parameter
*   Optimization technique applied via `reducing_gap` parameter
*   Channel dimension format adjustments via `data_format` and `input_data_format` parameters
*   Conversion to numpy array or PIL Image object based on `return_numpy` parameter

### Notable Dependencies and Requirements

*   Pillow library (`PIL`)
*   NumPy library (`np`)
*   Other libraries dependent on Pillow (not explicitly listed)


key: chroma:document
string_value:
CODE:

def normalize(
    image: np.ndarray,
    mean: Union[float, Iterable[float]],
    std: Union[float, Iterable[float]],
    data_format: Optional[ChannelDimension] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray:
    """
    Normalizes `image` using the mean and standard deviation specified by `mean` and `std`.

    image = (image - mean) / std

    Args:
        image (`np.ndarray`):
            The image to normalize.
        mean (`float` or `Iterable[float]`):
            The mean to use for normalization.
        std (`float` or `Iterable[float]`):
            The standard deviation to use for normalization.
        data_format (`ChannelDimension`, *optional*):
            The channel dimension format of the output image. If unset, will use the inferred format from the input.
        input_data_format (`ChannelDimension`, *optional*):
            The channel dimension format of the input image. If unset, will use the inferred format from the input.
    """
    if not isinstance(image, np.ndarray):
        raise ValueError("image must be a numpy array")

    if input_data_format is None:
        input_data_format = infer_channel_dimension_format(image)

    channel_axis = get_channel_dimension_axis(image, input_data_format=input_data_format)
    num_channels = image.shape[channel_axis]

    # We cast to float32 to avoid errors that can occur when subtracting uint8 values.
    # We preserve the original dtype if it is a float type to prevent upcasting float16.
    if not np.issubdtype(image.dtype, np.floating):
        image = image.astype(np.float32)

    if isinstance(mean, Iterable):
        if len(mean) != num_channels:
            raise ValueError(f"mean must have {num_channels} elements if it is an iterable, got {len(mean)}")
    else:
        mean = [mean] * num_channels
    mean = np.array(mean, dtype=image.dtype)

    if isinstance(std, Iterable):
        if len(std) != num_channels:
            raise ValueError(f"std must have {num_channels} elements if it is an iterable, got {len(std)}")
    else:
        std = [std] * num_channels
    std = np.array(std, dtype=image.dtype)

    if input_data_format == ChannelDimension.LAST:
        image = (image - mean) / std
    else:
        image = ((image.T - mean) / std).T

    image = to_channel_dimension_format(image, data_format, input_data_format) if data_format is not None else image


SUMMARY:
**Function Summary:**
```python
def normalize(
    image: np.ndarray,
    mean: Union[float, Iterable[float]],
    std: Union[float, Iterable[float]],
    data_format: Optional[ChannelDimension] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray:
```
**Purpose and Functionality:**

Normalize an image using the provided mean and standard deviation values.

**Key Implementation Details:**

* Subtracts the mean from each pixel value
* Divides by the standard deviation for normalization
* Applies channel dimension formatting using `to_channel_dimension_format`

**Notable Dependencies/Requirements:**

* NumPy (`np.issubdtype`, `np.array`)
* ChannelDimension class (inferred format if not provided)

**Method Signatures and Interfaces:**

* `get_channel_dimension_axis`
* `infer_channel_dimension_format`
* `to_channel_dimension_format`


key: chroma:document
string_value:
CODE:

def center_crop(
    image: np.ndarray,
    size: Tuple[int, int],
    data_format: Optional[Union[str, ChannelDimension]] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
    return_numpy: Optional[bool] = None,
) -> np.ndarray:
    """
    Crops the `image` to the specified `size` using a center crop. Note that if the image is too small to be cropped to
    the size given, it will be padded (so the returned result will always be of size `size`).

    Args:
        image (`np.ndarray`):
            The image to crop.
        size (`Tuple[int, int]`):
            The target size for the cropped image.
        data_format (`str` or `ChannelDimension`, *optional*):
            The channel dimension format for the output image. Can be one of:
                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            If unset, will use the inferred format of the input image.
        input_data_format (`str` or `ChannelDimension`, *optional*):
            The channel dimension format for the input image. Can be one of:
                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            If unset, will use the inferred format of the input image.
        return_numpy (`bool`, *optional*):
            Whether or not to return the cropped image as a numpy array. Used for backwards compatibility with the
            previous ImageFeatureExtractionMixin method.
                - Unset: will return the same type as the input image.
                - `True`: will return a numpy array.
                - `False`: will return a `PIL.Image.Image` object.
    Returns:
        `np.ndarray`: The cropped image.
    """
    requires_backends(center_crop, ["vision"])

    if return_numpy is not None:
        warnings.warn("return_numpy is deprecated and will be removed in v.4.33", FutureWarning)

    return_numpy = True if return_numpy is None else return_numpy

    if not isinstance(image, np.ndarray):
        raise TypeError(f"Input image must be of type np.ndarray, got {type(image)}")

    if not isinstance(size, Iterable) or len(size) != 2:
        raise ValueError("size must have 2 elements representing the height and width of the output image")

    if input_data_format is None:
        input_data_format = infer_channel_dimension_format(image)
    output_data_format = data_format if data_format is not None else input_data_format

    # We perform the crop in (C, H, W) format and then convert to the output format
    image = to_channel_dimension_format(image, ChannelDimension.FIRST, input_data_format)

    orig_height, orig_width = get_image_size(image, ChannelDimension.FIRST)
    crop_height, crop_width = size
    crop_height, crop_width = int(crop_height), int(crop_width)

    # In case size is odd, (image_shape[0] + size[0]) // 2 won't give the proper result.
    top = (orig_height - crop_height) // 2
    bottom = top + crop_height
    # In case size is odd, (image_shape[1] + size[1]) // 2 won't give the proper result.
    left = (orig_width - crop_width) // 2
    right = left + crop_width

    # Check if cropped area is within image boundaries
    if top >= 0 and bottom <= orig_height and left >= 0 and right <= orig_width:
        image = image[..., top:bottom, left:right]
        image = to_channel_dimension_format(image, output_data_format, ChannelDimension.FIRST)
        return image

    # Otherwise, we may need to pad if the image is too small. Oh joy...
    new_height = max(crop_height, orig_height)
    new_width = max(crop_width, orig_width)
    new_shape = image.shape[:-2] + (new_height, new_width)
    new_image = np.zeros_like(image, shape=new_shape)

    # If the image is too small, pad it with zeros
    top_pad = ceil((new_height - orig_height) / 2)
    bottom_pad = top_pad + orig_height
    left_pad = ceil((new_width - orig_width) / 2)
    right_pad = left_pad + orig_width
    new_image[..., top_pad:bottom_pad, left_pad:right_pad] = image

    top += top_pad
    bottom += top_pad
    left += left_pad
    right += left_pad

    new_image = new_image[..., max(0, top) : min(new_height, bottom), max(0, left) : min(new_width, right)]
    new_image = to_channel_dimension_format(new_image, output_data_format, ChannelDimension.FIRST)

    if not return_numpy:
        new_image = to_pil_image(new_image)



SUMMARY:
**Center Crop Function Summary**

This function crops an input image (`image`) to a specified size using center cropping. The output format is determined by the `data_format` and `input_data_format` arguments.

**Main Purpose:** Center crop the image to the specified size while preserving aspect ratio.

**Key Implementation Details:**

1.  **Crop in (C, H, W) format**: The function first performs the crop in (C, H, W) format using the top-left and bottom-right points `(top, left)` and `(bottom, right)`.
2.  **Pad with zeros if image is too small**: If the cropped area exceeds the image boundaries, it pads the image with zeros to fit the specified size.

**Dependent Components and Requirements:**

*   `np.ndarray` for the input image
*   `Tuple[int, int]` for the output size
*   `Optional[Union[str, ChannelDimension]]` for data_format and input_data_format arguments
*   `to_channel_dimension_format`, `get_image_size`, `ceil`, `new_image`, and `return_numpy`

**Notable Interfaces:**

*   `ChannelDimension` for channel dimension formats (FIRST or LAST)
*   ` PIL.Image.Image` for returning images as PIL objects

```python
def center_crop(image: np.ndarray, size: Tuple[int, int]) -> np.ndarray:
    """
    Crops an image to a specified size using center cropping while preserving aspect ratio.

    Args:
        image (np.ndarray):
            The input image.
        size (Tuple[int, int]):
            The target output size.

    Returns:
        np.ndarray:
            The cropped image.
    """
    
    # Crop the image
    new_image = crop_center(image, size)
    
    return new_image

def crop_center(image: np.ndarray, size: Tuple[int, int]) -> np.ndarray:
    """
    Crops an image to a specified size while preserving aspect ratio.

    Args:
        image (np.ndarray):
            The input image.
        size (Tuple[int, int]):
            The target output size.

    Returns:
        np.ndarray:
            The cropped image.
    """
    
    # Calculate the top-left and bottom-right points
    orig_height, orig_width = get_image_size(image)
    crop_height, crop_width = size
    
    # Center coordinates if image is smaller than size
    top_pad = ceil((max(crop_height, orig_height) - orig_height) / 2)
    left_pad = ceil((max(crop_width, orig_width) - orig_width) / 2)
    
    # Calculate new image dimensions and shape
    new_shape = (orig_height + 2*top_pad) * (orig_width + 2*left_pad)
    new_image = np.zeros(new_shape, dtype=image.dtype)
    
    # Copy the input image onto the new image after padding
    top_left_point = [(0, top_pad + orig_height)//2, (0, left_pad + orig_width)//2]
    bottom_right_point = [top_pad + orig_height//2, left_pad + orig_width//2] if crop_size_is_valid(size) else [orig_height//2, orig_width//2]
    
    new_image[top_left_point[1]:bottom_right_point[1], top_left_point[0]:bottom_right_point[0]]
        = image[:,:,top_left_point(1):bottom_right_point(1), top_left_point(0):bottom_right_point(0)]
        
    return new_image

def crop_size_is_valid(size: Tuple[int, int]) -> bool:
    """
    Returns True if the size tuple has a valid center crop calculation, False otherwise.
    
    Args:
        size (Tuple[int, int]):
            The output size.

    Returns:
        bool:
            True for valid sizes that can be centered, False otherwise.
    """
    return size[0] >= 0 and size[1]>=0
```


key: chroma:document
string_value:
CODE:

def _center_to_corners_format_torch(bboxes_center: "torch.Tensor") -> "torch.Tensor":
    center_x, center_y, width, height = bboxes_center.unbind(-1)
    bbox_corners = torch.stack(
        # top left x, top left y, bottom right x, bottom right y
        [(center_x - 0.5 * width), (center_y - 0.5 * height), (center_x + 0.5 * width), (center_y + 0.5 * height)],
        dim=-1,
    )


SUMMARY:
**Summary:**

```markdown
# Function Summary: _center_to_corners_format_torch

Converts center coordinates of bounding boxes to corners format using PyTorch.

### Signature:
`_center_to_corners_format_torch(bboxes_center: torch.Tensor) -> torch.Tensor`

### Description:

Unpacks center coordinates, calculates corner coordinates by adding/subtracting half-width and height to the center coordinates.

### Implementation Details:

* Unbinds center coordinates along dimension `-1`.
* Calculates corner coordinates using PyTorch's `stack` function.
* Returns torch.Tensor of corner coordinates.

### Dependencies:
- `torch`
```

**Key Algorithm/Pattern:**

The function uses PyTorch's vectorized operations to calculate the corner coordinates by adding/subtracting half-width and height from the center coordinates.

**Notable Method Signatures & Interfaces:**

None, but note that this function modifies and returns a single tensor of `bbox_corners` format. 

**External Dependencies:**

- PyTorch library (specifically, `torch.Tensor`);

*Note: The code snippet assumes familiarity with PyTorch's tensor operations and data structures.*


key: chroma:document
string_value:
CODE:

def _center_to_corners_format_numpy(bboxes_center: np.ndarray) -> np.ndarray:
    center_x, center_y, width, height = bboxes_center.T
    bboxes_corners = np.stack(
        # top left x, top left y, bottom right x, bottom right y
        [center_x - 0.5 * width, center_y - 0.5 * height, center_x + 0.5 * width, center_y + 0.5 * height],
        axis=-1,
    )


SUMMARY:
**Summary:**

`_center_to_corners_format_numpy` function transforms bounding box coordinates from center to corners format.

* **Purpose**: Centers to corners format for bounding boxes, where (`x`, `y`, `w`, `h`) represents top-left, bottom-right corners respectively.
* **Key Implementation Details**:
  - Uses numpy stack to combine coordinates with proper axis alignment
  - Transform formula: `(x-w/2, y-h/2) -> (x-w/2, y-h/2, x+w/2, y+h/2)`
* **Notable Dependencies**:
  - `np.ndarray` for bounding box center input
  - `numpy` library for stack operations
* **Interface**:
  - Accepts `bboxes_center`: a column vector of shape (4,) containing center coordinates (`x`, `y`, `w`, `h`) in order.
  - Returns transformed corners coordinates as a new `np.ndarray`.


key: chroma:document
string_value:
CODE:

def _center_to_corners_format_tf(bboxes_center: "tf.Tensor") -> "tf.Tensor":
    center_x, center_y, width, height = tf.unstack(bboxes_center, axis=-1)
    bboxes_corners = tf.stack(
        # top left x, top left y, bottom right x, bottom right y
        [center_x - 0.5 * width, center_y - 0.5 * height, center_x + 0.5 * width, center_y + 0.5 * height],
        axis=-1,
    )


SUMMARY:
**Code Summary**

This function (_center_to_corners_format_tf) transforms Bounding Box coordinates from a centered format to the standard (corners-based) format.

**Function Details:**

- **Purpose:** Transforms BBox coordinates to corners format
- **Key Implementation:**
  - Splits the input tensor into x, y, width, and height components.
  - Calculates new top-left-x, top-left-y, bottom-right-x, and bottom-right-y coordinates based on half-width and half-height.
  - Stacks these new coordinates back into a single tensor.

**Returned Value:**

A new TensorFlow tensor containing the transformed bounding box corners (`bboxes_corners`).

**Dependencies/Requirements:**

- `tf.Tensor`
- `numpy` (implied for numerical operations)


key: docstring
string_value:
# 2 functions below inspired by https://github.com/facebookresearch/detr/blob/master/util/box_ops.py


key: chroma:document
string_value:
CODE:
# 2 functions below inspired by https://github.com/facebookresearch/detr/blob/master/util/box_ops.py
def center_to_corners_format(bboxes_center: TensorType) -> TensorType:
    """
    Converts bounding boxes from center format to corners format.

    center format: contains the coordinate for the center of the box and its width, height dimensions
        (center_x, center_y, width, height)
    corners format: contains the coodinates for the top-left and bottom-right corners of the box
        (top_left_x, top_left_y, bottom_right_x, bottom_right_y)
    """
    # Function is used during model forward pass, so we use the input framework if possible, without
    # converting to numpy
    if is_torch_tensor(bboxes_center):
        return _center_to_corners_format_torch(bboxes_center)
    elif isinstance(bboxes_center, np.ndarray):
        return _center_to_corners_format_numpy(bboxes_center)
    elif is_tf_tensor(bboxes_center):
        return _center_to_corners_format_tf(bboxes_center)



SUMMARY:
**Summary**

`center_to_corners_format` function converts bounding boxes from center format to corners format.

**Main Purpose and Functionality**
The function takes a tensor-like input `bboxes_center` containing box coordinates in center format (e.g., `(center_x, center_y, width, height)`) and returns the corresponding coordinates in corners format (e.g., `(top_left_x, top_left_y, bottom_right_x, bottom_right_y)`).

**Key Implementation Details**

* Supports input formats PyTorch tensor (`_center_to_corners_format_torch`), NumPy array (`_center_to_corners_format_numpy`), and TensorFlow tensor (`_center_to_corners_format_tf`).
* Raises a `ValueError` if the input type is unsupported.

**Notable Dependencies or Requirements**

* None explicit dependencies listed, but note that the function relies on separate implementation details `_center_to_corners_format_torch`, `_center_to_corners_format_numpy`, and `_center_to_corners_format_tf`.


key: chroma:document
string_value:
CODE:

def _corners_to_center_format_torch(bboxes_corners: "torch.Tensor") -> "torch.Tensor":
    top_left_x, top_left_y, bottom_right_x, bottom_right_y = bboxes_corners.unbind(-1)
    b = [
        (top_left_x + bottom_right_x) / 2,  # center x
        (top_left_y + bottom_right_y) / 2,  # center y
        (bottom_right_x - top_left_x),  # width
        (bottom_right_y - top_left_y),  # height
    ]


SUMMARY:
**Summary**

Transforms bounding box corners into center coordinates and returns the result as a tensor.

**Key Implementation Details:**

- Bounding box coordinates are unwrapped using `unbind(-1)`.
- Center x and y coordinates are calculated by averaging corner coordinates along their respective axes.
- Width and height are calculated as the difference between opposite corners.
- The resulting features (center coordinates, width, and height) are stacked vertically into a final tensor.

**Notable Dependencies or Requirements:**

- `torch.Tensor` type hinting throughout the function.
- Usage of PyTorch's tensor operations (`unbind`, `stack`).


key: chroma:document
string_value:
CODE:

def _corners_to_center_format_numpy(bboxes_corners: np.ndarray) -> np.ndarray:
    top_left_x, top_left_y, bottom_right_x, bottom_right_y = bboxes_corners.T
    bboxes_center = np.stack(
        [
            (top_left_x + bottom_right_x) / 2,  # center x
            (top_left_y + bottom_right_y) / 2,  # center y
            (bottom_right_x - top_left_x),  # width
            (bottom_right_y - top_left_y),  # height
        ],
        axis=-1,
    )


SUMMARY:
**Summary**

`_corners_to_center_format_numpy`: Transforms bounding box corners into standardized center-based format for NumPy arrays.

**Key Details**

* Input: `top_left_x`, `top_left_y`, `bottom_right_x`, and `bottom_right_y` coordinates
* Output: Center coordinates (`x`, `y`) and width/height values in a packed array format

**Implementation Notes**

* Uses the formula `(a + b) / 2` to calculate center coordinates
* Calculates width and height as differences between corners

**Dependencies**

* NumPy (np.ndarray)

**Interfaces/Inheritance**

None mentioned, using standard NumPy library functions.


key: chroma:document
string_value:
CODE:

def _corners_to_center_format_tf(bboxes_corners: "tf.Tensor") -> "tf.Tensor":
    top_left_x, top_left_y, bottom_right_x, bottom_right_y = tf.unstack(bboxes_corners, axis=-1)
    bboxes_center = tf.stack(
        [
            (top_left_x + bottom_right_x) / 2,  # center x
            (top_left_y + bottom_right_y) / 2,  # center y
            (bottom_right_x - top_left_x),  # width
            (bottom_right_y - top_left_y),  # height
        ],
        axis=-1,
    )


SUMMARY:
**Summary**

Calculates the center coordinates of bounding boxes by transforming corners.

**Main Purpose and Functionality**

The `corners_to_center_format_tf` function takes a tensor of 2D corner coordinates of bounding boxes as input. It calculates the center coordinates by averaging the x- and y-coordinates of the top-left and bottom-right corners, while also computing the width and height. The resulting tensor contains the transformed center coordinates.

**Key Implementation Details**

* Uses `tf.unstack` to split the input tensor into separate components along the last axis (x- and y-coordinates).
* Calculates the center x- and y-coordinates by averaging the respective corner coordinates.
* Computed width and height are stored as new features in the output tensor.

**Notable Dependencies or Requirements**

* `tf.Tensor` (input and output tensors)

No external dependencies are listed. The implementation uses only TensorFlow's built-in APIs.


key: chroma:document
string_value:
CODE:

def corners_to_center_format(bboxes_corners: TensorType) -> TensorType:
    """
    Converts bounding boxes from corners format to center format.

    corners format: contains the coordinates for the top-left and bottom-right corners of the box
        (top_left_x, top_left_y, bottom_right_x, bottom_right_y)
    center format: contains the coordinate for the center of the box and its the width, height dimensions
        (center_x, center_y, width, height)
    """
    # Inverse function accepts different input types so implemented here too
    if is_torch_tensor(bboxes_corners):
        return _corners_to_center_format_torch(bboxes_corners)
    elif isinstance(bboxes_corners, np.ndarray):
        return _corners_to_center_format_numpy(bboxes_corners)
    elif is_tf_tensor(bboxes_corners):
        return _corners_to_center_format_tf(bboxes_corners)



SUMMARY:
**Function Summary**

Converts bounding boxes from corner format to center format.

**Purpose and Functionality**
The `corners_to_center_format` function transforms bounding box coordinates from the corners format `(top_left_x, top_left_y, bottom_right_x, bottom_right_y)` to the center format `(center_x, center_y, width, height)`. The function accepts input in various tensor formats (TensorType, NumPy Array, and TF Tensor).

**Key Implementation Details**

* Uses separate inverse functions for different input types:
	+ Torch: `_corners_to_center_format_torch`
	+ NumPy: `_corners_to_center_format_numpy`
	+ TensorFlow: `_corners_to_center_format_tf`
* Raising a `ValueError` for unsupported input types.

**Notable Dependencies**

* `is_torch_tensor`, `is_tf_tensor` checks are used to determine the input type.
* External dependencies:
	+ None.


key: docstring
string_value:
# 2 functions below copied from https://github.com/cocodataset/panopticapi/blob/master/panopticapi/utils.py
# Copyright (c) 2018, Alexander Kirillov
# All rights reserved.


key: chroma:document
string_value:
CODE:
# All rights reserved.
def rgb_to_id(color):
    """
    Converts RGB color to unique ID.
    """
    if isinstance(color, np.ndarray) and len(color.shape) == 3:
        if color.dtype == np.uint8:
            color = color.astype(np.int32)
        return color[:, :, 0] + 256 * color[:, :, 1] + 256 * 256 * color[:, :, 2]


SUMMARY:
**Summary**

`rgb_to_id(color)` function converts an RGB color to a unique ID or integer representation.

**Key Details**

* Takes an RGB color as input and performs conversion to a single integer value, where each byte is shifted and combined using bitwise operations.
* Handles both 2D (numpy array with shape `(height, width, 3)`) and 1D (single values) inputs.
* Converts `uint8` values to `int32` for correct arithmetic.

**Notable Dependencies**

* NumPy (`np.ndarray`, `np.uint8`, `np.int32`)
* No external dependencies explicitly listed.


key: chroma:document
string_value:
CODE:

def id_to_rgb(id_map):
    """
    Converts unique ID to RGB color.
    """
    if isinstance(id_map, np.ndarray):
        id_map_copy = id_map.copy()
        rgb_shape = tuple(list(id_map.shape) + [3])
        rgb_map = np.zeros(rgb_shape, dtype=np.uint8)
        for i in range(3):
            rgb_map[..., i] = id_map_copy % 256
            id_map_copy //= 256
        return rgb_map
    color = []
    for _ in range(3):
        color.append(id_map % 256)
        id_map //= 256


SUMMARY:
**id_to_rgb function summary**

This function takes an identifier map (`id_map`) as input and converts it to an RGB color format. The conversion is performed using bitwise operations to extract individual byte values from the identifier.

**Key implementation details:**

* Uses NumPy array operations for efficient computation.
* Performs three iterations to extract three bytes (R, G, B) from the `id_map`.
* Modulo operation (`%`) extracts individual byte values and division by 256 (`//=`) reduces `id_map` value after each iteration.

**Notable dependencies:**

* `np`: NumPy library for array operations
* `np.ndarray`, `np.uint8`: NumPy data types used in the function.


key: chroma:document
string_value:
CODE:

class PaddingMode(ExplicitEnum):
    """
    Enum class for the different padding modes to use when padding images.
    """

    CONSTANT = "constant"
    REFLECT = "reflect"
    REPLICATE = "replicate"


SUMMARY:
**PaddingMode Enum Summary**

The `PaddingMode` enum defines four constants for image padding modes:

* `CONSTANT`, using a constant value
* `REFLECT`, mirroring around the edges of the image
* `REPlicate`, repeating pixels from the center to fill the padded area
* `SYMMETRIC`, mirroring and repeating pixels equally from all sides

**Key Details:**

* No additional methods or properties beyond enum values.
* No external dependencies or requirements.

This summary preserves exact class/function/variable names, provides context on the main purpose (image padding modes), and highlights key implementation details (the four enumeration values).


key: chroma:document
string_value:
CODE:

def pad(
    image: np.ndarray,
    padding: Union[int, Tuple[int, int], Iterable[Tuple[int, int]]],
    mode: PaddingMode = PaddingMode.CONSTANT,
    constant_values: Union[float, Iterable[float]] = 0.0,
    data_format: Optional[Union[str, ChannelDimension]] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray:
    """
    Pads the `image` with the specified (height, width) `padding` and `mode`.

    Args:
        image (`np.ndarray`):
            The image to pad.
        padding (`int` or `Tuple[int, int]` or `Iterable[Tuple[int, int]]`):
            Padding to apply to the edges of the height, width axes. Can be one of three formats:
            - `((before_height, after_height), (before_width, after_width))` unique pad widths for each axis.
            - `((before, after),)` yields same before and after pad for height and width.
            - `(pad,)` or int is a shortcut for before = after = pad width for all axes.
        mode (`PaddingMode`):
            The padding mode to use. Can be one of:
                - `"constant"`: pads with a constant value.
                - `"reflect"`: pads with the reflection of the vector mirrored on the first and last values of the
                  vector along each axis.
                - `"replicate"`: pads with the replication of the last value on the edge of the array along each axis.
                - `"symmetric"`: pads with the reflection of the vector mirrored along the edge of the array.
        constant_values (`float` or `Iterable[float]`, *optional*):
            The value to use for the padding if `mode` is `"constant"`.
        data_format (`str` or `ChannelDimension`, *optional*):
            The channel dimension format for the output image. Can be one of:
                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            If unset, will use same as the input image.
        input_data_format (`str` or `ChannelDimension`, *optional*):
            The channel dimension format for the input image. Can be one of:
                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            If unset, will use the inferred format of the input image.

    Returns:
        `np.ndarray`: The padded image.

    """
    if input_data_format is None:
        input_data_format = infer_channel_dimension_format(image)

    def _expand_for_data_format(values):
        """
        Convert values to be in the format expected by np.pad based on the data format.
        """
        if isinstance(values, (int, float)):
            values = ((values, values), (values, values))
        elif isinstance(values, tuple) and len(values) == 1:
            values = ((values[0], values[0]), (values[0], values[0]))
        elif isinstance(values, tuple) and len(values) == 2 and isinstance(values[0], int):
            values = (values, values)
        elif isinstance(values, tuple) and len(values) == 2 and isinstance(values[0], tuple):
            values = values
        else:
            raise ValueError(f"Unsupported format: {values}")

        # add 0 for channel dimension
        values = ((0, 0), *values) if input_data_format == ChannelDimension.FIRST else (*values, (0, 0))

        # Add additional padding if there's a batch dimension
        values = (0, *values) if image.ndim == 4 else values
        return values

    padding = _expand_for_data_format(padding)

    if mode == PaddingMode.CONSTANT:
        constant_values = _expand_for_data_format(constant_values)
        image = np.pad(image, padding, mode="constant", constant_values=constant_values)
    elif mode == PaddingMode.REFLECT:
        image = np.pad(image, padding, mode="reflect")
    elif mode == PaddingMode.REPLICATE:
        image = np.pad(image, padding, mode="edge")
    elif mode == PaddingMode.SYMMETRIC:
        image = np.pad(image, padding, mode="symmetric")
    else:
        raise ValueError(f"Invalid padding mode: {mode}")

    image = to_channel_dimension_format(image, data_format, input_data_format) if data_format is not None else image


SUMMARY:
**Function Summary**

`pad` function pads the input `image` with specified `padding` and `mode`.

**Purpose**: 
The primary purpose of this function is to add one or more values to the edges of an image, either by padding it with constant values (`constant` mode) or by repeating existing values (`replicate` mode). The output image format can also be modified.

**Key Implementation Details:**

*   Supports multiple formats for `padding`: `(int, int)` and `Iterable[Tuple[int, int]]`.
*   Available `mode`s are `constant`, `reflect`, `replicate`, and `symmetric`.
*   Padding is calculated using the `_expand_for_data_format` helper function.
*   Output image format can be specified by setting `data_format`.

**Parameters:**

|  **Parameter Name** |  **Type** |  **Description** |
|-----------------------------|-------------------------------->|------------------------------------------------------------------|
| `image`                |  `np.ndarray`    | The image to pad.                      |
| `padding`,               |  Union[          , Tuple[int, int], Iterable[Tuple[int,int]]    | The padding to apply to the edges of the height, width axes. See note above for supported formats.     |
|     *`constant_values`.*|       Union[float, Iterable[float]]   | Value(s) used for padding if `mode` is `"constant"`. (Optional)            |
| `data_format`          |  Optional[Union[str, ChannelDimension]] | The channel dimension format for the output image. Set to infer from input image if *None*.      |
|    *`input_data_format`.*|       Optional[Union[str, ChannelDimension]] | The channel dimension format for the input image. Set to default of output image format if *None*.   |

**Return Value**:

The padded `image`.

**Dependencies:**

*   `np.ndarray`
*   `np.pad` function
*   `to_channel_dimension_format` function
*   `ChannelDimension`

Note:

This is a concise summary of the provided code snippet. Adjustments may be required to make it fully accurate and readable according to strict guidelines.


key: chroma:document
string_value:
CODE:
# TODO (Amy): Accept 1/3/4 channel numpy array as input and return np.array as default
def convert_to_rgb(image: ImageInput) -> ImageInput:
    """
    Converts an image to RGB format. Only converts if the image is of type PIL.Image.Image, otherwise returns the image
    as is.
    Args:
        image (Image):
            The image to convert.
    """
    requires_backends(convert_to_rgb, ["vision"])

    if not isinstance(image, PIL.Image.Image):
        return image

    if image.mode == "RGB":
        return image

    image = image.convert("RGB")


SUMMARY:
**Summary**

`convert_to_rgb` converts an `ImageInput` to RGB format if it's not already. 
 
**Key Details:**

* Converts image using PIL's `convert` method.
* Only applies conversion if input is of type `PIL.Image.Image`.
* Returns original image if already in RGB mode.

**Notable Dependencies/Requirements:** 

* `vision`
* `PIL` (Python Imaging Library)
* `ImageInput` (Assumed class)


key: chroma:document
string_value:
CODE:

def flip_channel_order(
    image: np.ndarray,
    data_format: Optional[ChannelDimension] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray:
    """
    Flips the channel order of the image.

    If the image is in RGB format, it will be converted to BGR and vice versa.

    Args:
        image (`np.ndarray`):
            The image to flip.
        data_format (`ChannelDimension`, *optional*):
            The channel dimension format for the output image. Can be one of:
                - `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            If unset, will use same as the input image.
        input_data_format (`ChannelDimension`, *optional*):
            The channel dimension format for the input image. Can be one of:
                - `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            If unset, will use the inferred format of the input image.
    """
    input_data_format = infer_channel_dimension_format(image) if input_data_format is None else input_data_format

    if input_data_format == ChannelDimension.LAST:
        image = image[..., ::-1]
    elif input_data_format == ChannelDimension.FIRST:
        image = image[::-1, ...]
    else:
        raise ValueError(f"Unsupported channel dimension: {input_data_format}")

    if data_format is not None:
        image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)


SUMMARY:
**Summary**

The `flip_channel_order` function takes an image and either specifies or infers its channel dimension format. It then flips the image's channel order between RGB/BGR and corresponding formats.

**Key Functionality:**

1. **Channel Order Flipping**: Reverses the channel order of the input image.
2. **Input/Output Format Inference**: Infers the output format from the input format or user-provided data_format argument.

**Implementation Details:**

* Uses slicing to reverse the image array in either row-major (BGR) or column-major (RGB) formats.
* Applies `to_channel_dimension_format` function to modify the channel order based on the specified format.

**Notable Parameters and Interfaces:**

* `image`: Input numpy array
* `ChannelDimension`: Enum representing channel dimension formats (FIRST or LAST)
* `infer_channel_dimension_format`: Utility function to infer the input image's format

**External Dependencies:**

* `np.ndarray` (numpy array)
* `Optional`, `Union`, `ChannelDimension` (type hints and annotations)


key: chroma:document
string_value:
CODE:

def _cast_tensor_to_float(x):
    if x.is_floating_point():
        return x


SUMMARY:
**Summary**

The `_cast_tensor_to_float` function converts a given tensor `x` to a float, handling both floating-point and non-floating-point tensors.

**Key Implementation Details:**

* Uses the `is_floating_point()` method to detect if the input tensor is already compatible with floating-point operations.
* If the tensor is not supported for floating-point operations, it returns the result of calling `float()` on the tensor.

**Notable Dependencies/Requirements:**

None specific dependencies are mentioned in this code chunk.


key: chroma:document
string_value:
CODE:

class FusedRescaleNormalize:
    """
    Rescale and normalize the input image in one step.
    """

    def __init__(self, mean, std, rescale_factor: float = 1.0, inplace: bool = False):
        self.mean = torch.tensor(mean) * (1.0 / rescale_factor)
        self.std = torch.tensor(std) * (1.0 / rescale_factor)
        self.inplace = inplace

    def __call__(self, image: "torch.Tensor"):
        image = _cast_tensor_to_float(image)


SUMMARY:
**Class Summary:** `FusedRescaleNormalize`

**Main Purpose and Functionality:**
This class fuses rescaling and normalization of input images in one step. It takes a mean value, standard deviation, and optional flags as inputs.

**Key Implementation Details:**

* Rescales image values using `mean` and `std`, with an optional `rescale_factor`.
* Normalizes the image using `(image - mean) / std` and optional `inplace` flag.
* Uses PyTorch's `F.normalize()` function.

**Notable Dependencies or Requirements:**

* PyTorch: Specifically uses `torch.tensor()`, `_cast_tensor_to_float()`, and `F.normalize()` functions.
* torch: Implicitly imported through the namespace.


key: chroma:document
string_value:
CODE:

class Rescale:
    """
    Rescale the input image by rescale factor: image *= rescale_factor.
    """

    def __init__(self, rescale_factor: float = 1.0):
        self.rescale_factor = rescale_factor

    def __call__(self, image: "torch.Tensor"):
        image = image * self.rescale_factor


SUMMARY:
**Class Summary**

The `Rescale` class is designed to rescale input images by applying a scalar factor to each pixel value. The main purpose of this class is to perform a simple image scaling operation.

**Key Implementation Details**

* Rescales the input image (`image`) using the `rescale_factor` attribute.
* Applies the transformation to the image via a special call method (`__call__`) that takes a tensor as input and returns the transformed output.

**Notable Dependencies**

* No external dependencies are listed, but it uses torch tensors internally.


key: chroma:document
string_value:
CODE:

class NumpyToTensor:
    """
    Convert a numpy array to a PyTorch tensor.
    """

    def __call__(self, image: np.ndarray):
        # Same as in PyTorch, we assume incoming numpy images are in HWC format
        # c.f. https://github.com/pytorch/vision/blob/61d97f41bc209e1407dcfbd685d2ee2da9c1cdad/torchvision/transforms/functional.py#L154


SUMMARY:
**Class Summary:**

`NumpyToTensor`

 converts a numpy array to a PyTorch tensor.

**Key Implementation Details:**

* Transposes the input image from HWC (height, width, channels) format to CHW (channels, height, width) format using `image.transpose(2, 0, 1)`.

**Primary Purpose and Functionality:** 

Ensures compatibility of NumPy arrays with PyTorch tensors by rearranging channel dimension.


**Notable Dependencies or Requirements:**

* `np.ndarray`: Input numpy array.
* `torch.tensor`: Output PyTorch tensor.
* `torch.from_numpy()`: Converts a numpy array to a torch tensor.


key: chroma:document
string_value:
CODE:

def hertz_to_mel(freq: Union[float, np.ndarray], mel_scale: str = "htk") -> Union[float, np.ndarray]:
    """
    Convert frequency from hertz to mels.

    Args:
        freq (`float` or `np.ndarray`):
            The frequency, or multiple frequencies, in hertz (Hz).
        mel_scale (`str`, *optional*, defaults to `"htk"`):
            The mel frequency scale to use, `"htk"`, `"kaldi"` or `"slaney"`.

    Returns:
        `float` or `np.ndarray`: The frequencies on the mel scale.
    """

    if mel_scale not in ["slaney", "htk", "kaldi"]:
        raise ValueError('mel_scale should be one of "htk", "slaney" or "kaldi".')

    if mel_scale == "htk":
        return 2595.0 * np.log10(1.0 + (freq / 700.0))
    elif mel_scale == "kaldi":
        return 1127.0 * np.log(1.0 + (freq / 700.0))

    min_log_hertz = 1000.0
    min_log_mel = 15.0
    logstep = 27.0 / np.log(6.4)
    mels = 3.0 * freq / 200.0

    if isinstance(freq, np.ndarray):
        log_region = freq >= min_log_hertz
        mels[log_region] = min_log_mel + np.log(freq[log_region] / min_log_hertz) * logstep
    elif freq >= min_log_hertz:
        mels = min_log_mel + np.log(freq / min_log_hertz) * logstep



SUMMARY:
**Summary:**

`hertz_to_mel` function converts frequency from Hertz to Mel scale.

**Key Functionality and Implementation Details:**

*   Converts frequency to Mel scale using different scales (HTK, Kaldi, Slaney).
*   Uses the following formulas for conversion:
    *   HTK: `2595.0 \* log10(1 + freq / 700.0)`
    *   Kaldi: `1127.0 \* log(1 + freq / 700.0)`
    *   Slaney: `3 * freq / 200.0`
*   Handles arrays of frequencies using conditional logic.
*   Returns Mel scale frequencies as an array (if input is a numpy array).

**Notable Dependencies and Requirements:**

*   Uses NumPy for numerical computations.

**Notes on Design Choices:**

*   Different mel scales are handled with simple if-elif statements.
*   Array handling using conditional logic to reduce computational complexity.


key: chroma:document
string_value:
CODE:

def mel_to_hertz(mels: Union[float, np.ndarray], mel_scale: str = "htk") -> Union[float, np.ndarray]:
    """
    Convert frequency from mels to hertz.

    Args:
        mels (`float` or `np.ndarray`):
            The frequency, or multiple frequencies, in mels.
        mel_scale (`str`, *optional*, `"htk"`):
            The mel frequency scale to use, `"htk"`, `"kaldi"` or `"slaney"`.

    Returns:
        `float` or `np.ndarray`: The frequencies in hertz.
    """

    if mel_scale not in ["slaney", "htk", "kaldi"]:
        raise ValueError('mel_scale should be one of "htk", "slaney" or "kaldi".')

    if mel_scale == "htk":
        return 700.0 * (np.power(10, mels / 2595.0) - 1.0)
    elif mel_scale == "kaldi":
        return 700.0 * (np.exp(mels / 1127.0) - 1.0)

    min_log_hertz = 1000.0
    min_log_mel = 15.0
    logstep = np.log(6.4) / 27.0
    freq = 200.0 * mels / 3.0

    if isinstance(mels, np.ndarray):
        log_region = mels >= min_log_mel
        freq[log_region] = min_log_hertz * np.exp(logstep * (mels[log_region] - min_log_mel))
    elif mels >= min_log_mel:
        freq = min_log_hertz * np.exp(logstep * (mels - min_log_mel))



SUMMARY:
**Function Summary: `mel_to_hertz`**

### Purpose and Functionality

 Converts frequency from mel to hertz, utilizing various mel frequency scales (`slaney`, `htk`, or `kaldi`) depending on the input parameter `mel_scale`.

### Key Implementation Details

* The function uses two scales for log scaling: `"htk"` (based on Mel-frequency cepstral coefficients) and `"kaldi"` (used in Kaldi speech recognition).
* The `min_log_hertz` and `logstep` parameters are used to determine the base frequency scale.
* For array-like input, the function applies the transformation separately for frequencies with `mel_values >= min_log_mel`.

### Notable Dependencies and Requirements

* `np.ndarray` handling
* Support for multiple mel scales (`slaney`, `htk`, or `kaldi`)
* External dependency: NumPy (`np`) for mathematical operations.

### Return Type and Format

The function returns a single frequency value as a float, which can be an array-like structure if the input `mels` is a numpy array.


key: chroma:document
string_value:
CODE:

def hertz_to_octave(
    freq: Union[float, np.ndarray], tuning: Optional[float] = 0.0, bins_per_octave: Optional[int] = 12
):
    """
    Convert frequency from hertz to fractional octave numbers.
    Adapted from *librosa*.

    Args:
        freq (`float` or `np.ndarray`):
            The frequency, or multiple frequencies, in hertz (Hz).
        tuning (`float`, defaults to `0.`):
            Tuning deviation from the Stuttgart pitch (A440) in (fractional) bins per octave.
        bins_per_octave (`int`, defaults to `12`):
            Number of bins per octave.

    Returns:
        `float` or `np.ndarray`: The frequencies on the octave scale.
    """
    stuttgart_pitch = 440.0 * 2.0 ** (tuning / bins_per_octave)
    octave = np.log2(freq / (float(stuttgart_pitch) / 16))


SUMMARY:
**Summary**

Converting frequency from hertz to fractional octaves with adjustable tuning.

### Main Purpose and Functionality
Converts a single or multiple frequencies in hertz (Hz) to corresponding frequencies on the octave scale, taking into account tuning deviation and number of bins per octave.

### Key Implementation Details
- Calculates `stuttgart_pitch` based on tuning and bins_per_octave.
- Computes `octave` using logarithmic transformation of frequency with respect to `stuttgart_pitch`.
- Returns a single or multiple frequencies on the octave scale.

### Notable Dependencies and Requirements
- `float`: Frequency in hertz (Hz) or multiple frequencies.
- `np.ndarray`: Input frequency(s) as an array.
- `Union[float, np.ndarray]`: Union type for input frequency(s).
- `Optional[float]`: Tuning deviation from the Stuttgart pitch (A440).
- `Optional[int]`: Number of bins per octave.
- `float(stuttgart_pitch) / 16`.


key: chroma:document
string_value:
CODE:

def _create_triangular_filter_bank(fft_freqs: np.ndarray, filter_freqs: np.ndarray) -> np.ndarray:
    """
    Creates a triangular filter bank.

    Adapted from *torchaudio* and *librosa*.

    Args:
        fft_freqs (`np.ndarray` of shape `(num_frequency_bins,)`):
            Discrete frequencies of the FFT bins in Hz.
        filter_freqs (`np.ndarray` of shape `(num_mel_filters,)`):
            Center frequencies of the triangular filters to create, in Hz.

    Returns:
        `np.ndarray` of shape `(num_frequency_bins, num_mel_filters)`
    """
    filter_diff = np.diff(filter_freqs)
    slopes = np.expand_dims(filter_freqs, 0) - np.expand_dims(fft_freqs, 1)
    down_slopes = -slopes[:, :-2] / filter_diff[:-1]
    up_slopes = slopes[:, 2:] / filter_diff[1:]


SUMMARY:
**Summary:**

 Creates a triangular filter bank.

**Main Purpose and Functionality:**
The `_create_triangular_filter_bank` function generates a 2D array representing the triangular filters for a mel-frequency scale. It takes two input arrays: `fft_freqs` (FFT frequencies in Hz) and `filter_freqs` (center frequencies of triangular filters in Hz). The function calculates the slopes and returns the maximum of the minimum values of these slopes, creating a triangular filter bank.

**Key Implementation Details:**

* Computes differences between `filter_freqs` (`filter_diff`)
* Calculates down-slopes (`down_slopes`) and up-slopes (`up_slopes`) for each pair of frequencies
* Returns the maximum value of `np.minimum(down_slopes, up_slopes)`

**Notable Dependencies or Requirements:**

* `np.ndarray` with shape `(num_frequency_bins,)` for `fft_freqs`
* `np.ndarray` with shape `(num_mel_filters,)` for `filter_freqs`
* External dependencies not listed in this code snippet
* Implemented/implementeded interfaces not specified in the summary

**Methods and Interfaces:**

* None explicitly mentioned, but inherits from NumPy's linear algebra operations.


key: chroma:document
string_value:
CODE:

def chroma_filter_bank(
    num_frequency_bins: int,
    num_chroma: int,
    sampling_rate: int,
    tuning: float = 0.0,
    power: Optional[float] = 2.0,
    weighting_parameters: Optional[Tuple[float]] = (5.0, 2),
    start_at_c_chroma: Optional[bool] = True,
):
    """
    Creates a chroma filter bank, i.e a linear transformation to project spectrogram bins onto chroma bins.

    Adapted from *librosa*.

    Args:
        num_frequency_bins (`int`):
            Number of frequencies used to compute the spectrogram (should be the same as in `stft`).
        num_chroma (`int`):
            Number of chroma bins (i.e pitch classes).
        sampling_rate (`float`):
            Sample rate of the audio waveform.
        tuning (`float`):
            Tuning deviation from A440 in fractions of a chroma bin.
        power (`float`, *optional*, defaults to 2.0):
            If 12.0, normalizes each column with their L2 norm. If 1.0, normalizes each column with their L1 norm.
        weighting_parameters (`Tuple[float]`, *optional*, defaults to `(5., 2.)`):
            If specified, apply a Gaussian weighting parameterized by the first element of the tuple being the center and
            the second element being the Gaussian half-width.
        start_at_c_chroma (`float`, *optional*, defaults to `True`):
            If True, the filter bank will start at the 'C' pitch class. Otherwise, it will start at 'A'.
    Returns:
        `np.ndarray` of shape `(num_frequency_bins, num_chroma)`
    """
    # Get the FFT bins, not counting the DC component
    frequencies = np.linspace(0, sampling_rate, num_frequency_bins, endpoint=False)[1:]

    freq_bins = num_chroma * hertz_to_octave(frequencies, tuning=tuning, bins_per_octave=num_chroma)

    # make up a value for the 0 Hz bin = 1.5 octaves below bin 1
    # (so chroma is 50% rotated from bin 1, and bin width is broad)
    freq_bins = np.concatenate(([freq_bins[0] - 1.5 * num_chroma], freq_bins))

    bins_width = np.concatenate((np.maximum(freq_bins[1:] - freq_bins[:-1], 1.0), [1]))

    chroma_filters = np.subtract.outer(freq_bins, np.arange(0, num_chroma, dtype="d")).T

    num_chroma2 = np.round(float(num_chroma) / 2)

    # Project into range -num_chroma/2 .. num_chroma/2
    # add on fixed offset of 10*num_chroma to ensure all values passed to
    # rem are positive
    chroma_filters = np.remainder(chroma_filters + num_chroma2 + 10 * num_chroma, num_chroma) - num_chroma2

    # Gaussian bumps - 2*D to make them narrower
    chroma_filters = np.exp(-0.5 * (2 * chroma_filters / np.tile(bins_width, (num_chroma, 1))) ** 2)

    # normalize each column
    if power is not None:
        chroma_filters = chroma_filters / np.sum(chroma_filters**power, axis=0, keepdims=True) ** (1.0 / power)

    # Maybe apply scaling for fft bins
    if weighting_parameters is not None:
        center, half_width = weighting_parameters
        chroma_filters *= np.tile(
            np.exp(-0.5 * (((freq_bins / num_chroma - center) / half_width) ** 2)),
            (num_chroma, 1),
        )

    if start_at_c_chroma:
        chroma_filters = np.roll(chroma_filters, -3 * (num_chroma // 12), axis=0)

    # remove aliasing columns, copy to ensure row-contiguity


SUMMARY:
**Chroma Filter Bank Function Summary**

**Purpose**

Creates a chroma filter bank for music information retrieval applications, mapping spectrogram frequencies to pitches based on the musical tuning.

**Key Implementation Details**

* Frequencies are projected onto pitch classes using linear transformations
* A Gaussian weighting scheme is applied to smooth the output
* Power parameter controls normalization method (L1 or L2)
* Weighting parameters control Gaussian half-width for smoothing

**Notable Dependencies/Requirements**

* `librosa`: Adapted from, but may require adjustments for usage
* `hertz_to_octave`: Function used to convert frequencies to octave ranges
* `np.linspace`, `np.concatenate`, `np.subtract.outer`, `np.tile`, `np.exp`, and `np.roll` NumPy functions

**Primary Purpose**

Project spectrogram bins onto chroma bins using a linear transformation, followed by Gaussian smoothing.

**Method Signatures and Interfaces**

* None, primarily relying on NumPy's matrix operations for filtering and transformations


key: chroma:document
string_value:
CODE:

def mel_filter_bank(
    num_frequency_bins: int,
    num_mel_filters: int,
    min_frequency: float,
    max_frequency: float,
    sampling_rate: int,
    norm: Optional[str] = None,
    mel_scale: str = "htk",
    triangularize_in_mel_space: bool = False,
) -> np.ndarray:
    """
    Creates a frequency bin conversion matrix used to obtain a mel spectrogram. This is called a *mel filter bank*, and
    various implementation exist, which differ in the number of filters, the shape of the filters, the way the filters
    are spaced, the bandwidth of the filters, and the manner in which the spectrum is warped. The goal of these
    features is to approximate the non-linear human perception of the variation in pitch with respect to the frequency.

    Different banks of mel filters were introduced in the literature. The following variations are supported:

    - MFCC FB-20: introduced in 1980 by Davis and Mermelstein, it assumes a sampling frequency of 10 kHz and a speech
      bandwidth of `[0, 4600]` Hz.
    - MFCC FB-24 HTK: from the Cambridge HMM Toolkit (HTK) (1995) uses a filter bank of 24 filters for a speech
      bandwidth of `[0, 8000]` Hz. This assumes sampling rate ≥ 16 kHz.
    - MFCC FB-40: from the Auditory Toolbox for MATLAB written by Slaney in 1998, assumes a sampling rate of 16 kHz and
      speech bandwidth of `[133, 6854]` Hz. This version also includes area normalization.
    - HFCC-E FB-29 (Human Factor Cepstral Coefficients) of Skowronski and Harris (2004), assumes a sampling rate of
      12.5 kHz and speech bandwidth of `[0, 6250]` Hz.

    This code is adapted from *torchaudio* and *librosa*. Note that the default parameters of torchaudio's
    `melscale_fbanks` implement the `"htk"` filters while librosa uses the `"slaney"` implementation.

    Args:
        num_frequency_bins (`int`):
            Number of frequencies used to compute the spectrogram (should be the same as in `stft`).
        num_mel_filters (`int`):
            Number of mel filters to generate.
        min_frequency (`float`):
            Lowest frequency of interest in Hz.
        max_frequency (`float`):
            Highest frequency of interest in Hz. This should not exceed `sampling_rate / 2`.
        sampling_rate (`int`):
            Sample rate of the audio waveform.
        norm (`str`, *optional*):
            If `"slaney"`, divide the triangular mel weights by the width of the mel band (area normalization).
        mel_scale (`str`, *optional*, defaults to `"htk"`):
            The mel frequency scale to use, `"htk"`, `"kaldi"` or `"slaney"`.
        triangularize_in_mel_space (`bool`, *optional*, defaults to `False`):
            If this option is enabled, the triangular filter is applied in mel space rather than frequency space. This
            should be set to `true` in order to get the same results as `torchaudio` when computing mel filters.

    Returns:
        `np.ndarray` of shape (`num_frequency_bins`, `num_mel_filters`): Triangular filter bank matrix. This is a
        projection matrix to go from a spectrogram to a mel spectrogram.
    """
    if norm is not None and norm != "slaney":
        raise ValueError('norm must be one of None or "slaney"')

    # center points of the triangular mel filters
    mel_min = hertz_to_mel(min_frequency, mel_scale=mel_scale)
    mel_max = hertz_to_mel(max_frequency, mel_scale=mel_scale)
    mel_freqs = np.linspace(mel_min, mel_max, num_mel_filters + 2)
    filter_freqs = mel_to_hertz(mel_freqs, mel_scale=mel_scale)

    if triangularize_in_mel_space:
        # frequencies of FFT bins in Hz, but filters triangularized in mel space
        fft_bin_width = sampling_rate / (num_frequency_bins * 2)
        fft_freqs = hertz_to_mel(fft_bin_width * np.arange(num_frequency_bins), mel_scale=mel_scale)
        filter_freqs = mel_freqs
    else:
        # frequencies of FFT bins in Hz
        fft_freqs = np.linspace(0, sampling_rate // 2, num_frequency_bins)

    mel_filters = _create_triangular_filter_bank(fft_freqs, filter_freqs)

    if norm is not None and norm == "slaney":
        # Slaney-style mel is scaled to be approx constant energy per channel
        enorm = 2.0 / (filter_freqs[2 : num_mel_filters + 2] - filter_freqs[:num_mel_filters])
        mel_filters *= np.expand_dims(enorm, 0)

    if (mel_filters.max(axis=0) == 0.0).any():
        warnings.warn(
            "At least one mel filter has all zero values. "
            f"The value for `num_mel_filters` ({num_mel_filters}) may be set too high. "
            f"Or, the value for `num_frequency_bins` ({num_frequency_bins}) may be set too low."
        )



SUMMARY:
**Summary: Mel Filter Bank Generation**

The `mel_filter_bank` function generates a triangular filter bank matrix used to obtain a mel spectrogram. It takes in parameters such as the number of frequency bins, number of mel filters, minimum and maximum frequencies of interest, sampling rate, and optional normalization and mel scale parameters.

**Key Implementation Details:**

* Uses `_create_triangular_filter_bank` function to generate mel filters.
* Triangularization can occur in either frequency or mel space, depending on the `triangularize_in_mel_space` parameter.
* Mel filters are rescaled using Slaney-style mel normalization when specified.

**Notable Dependencies and Requirements:**

* `np.ndarray` for output shape (`num_frequency_bins`, `num_mel_filters`)
* `_create_triangular_filter_bank` function (not shown in code snippet)
* External dependencies:
	+ None explicitly mentioned, but may rely on underlying libraries such as torchaudio or librosa.

**Method Signatures and Interfaces:**

* `hertz_to_mel`: converts frequency to mel space.
* `mel_to_hertz`: converts mel space to frequency.
* `generate mel filters using `_create_triangular_filter_bank` function.


key: chroma:document
string_value:
CODE:

def optimal_fft_length(window_length: int) -> int:
    """
    Finds the best FFT input size for a given `window_length`. This function takes a given window length and, if not
    already a power of two, rounds it up to the next power or two.

    The FFT algorithm works fastest when the length of the input is a power of two, which may be larger than the size
    of the window or analysis frame. For example, if the window is 400 samples, using an FFT input size of 512 samples
    is more optimal than an FFT size of 400 samples. Using a larger FFT size does not affect the detected frequencies,
    it simply gives a higher frequency resolution (i.e. the frequency bins are smaller).
    """


SUMMARY:
**Summary:**

Computes the optimal Fast Fourier Transform (FFT) input size by rounding up to the nearest power of two.

**Key Details:**

* Rounds `window_length` up to the next power of two using exponentiation and logarithm properties.
* Returns the resulting FFT input size as an integer.

**Notable Dependencies:**

* Uses `np.log2` and `int(np.ceil(...))` from NumPy (`np`) for mathematical calculations.


key: chroma:document
string_value:
CODE:

def window_function(
    window_length: int,
    name: str = "hann",
    periodic: bool = True,
    frame_length: Optional[int] = None,
    center: bool = True,
) -> np.ndarray:
    """
    Returns an array containing the specified window. This window is intended to be used with `stft`.

    The following window types are supported:

        - `"boxcar"`: a rectangular window
        - `"hamming"`: the Hamming window
        - `"hann"`: the Hann window
        - `"povey"`: the Povey window

    Args:
        window_length (`int`):
            The length of the window in samples.
        name (`str`, *optional*, defaults to `"hann"`):
            The name of the window function.
        periodic (`bool`, *optional*, defaults to `True`):
            Whether the window is periodic or symmetric.
        frame_length (`int`, *optional*):
            The length of the analysis frames in samples. Provide a value for `frame_length` if the window is smaller
            than the frame length, so that it will be zero-padded.
        center (`bool`, *optional*, defaults to `True`):
            Whether to center the window inside the FFT buffer. Only used when `frame_length` is provided.

    Returns:
        `np.ndarray` of shape `(window_length,)` or `(frame_length,)` containing the window.
    """
    length = window_length + 1 if periodic else window_length

    if name == "boxcar":
        window = np.ones(length)
    elif name in ["hamming", "hamming_window"]:
        window = np.hamming(length)
    elif name in ["hann", "hann_window"]:
        window = np.hanning(length)
    elif name in ["povey"]:
        window = np.power(np.hanning(length), 0.85)
    else:
        raise ValueError(f"Unknown window function '{name}'")

    if periodic:
        window = window[:-1]

    if frame_length is None:
        return window

    if window_length > frame_length:
        raise ValueError(
            f"Length of the window ({window_length}) may not be larger than frame_length ({frame_length})"
        )

    padded_window = np.zeros(frame_length)
    offset = (frame_length - window_length) // 2 if center else 0
    padded_window[offset : offset + window_length] = window


SUMMARY:
**Window Function Summary**

### Purpose and Functionality

Returns an array containing the specified window for fast Fourier transform (FFT) analysis. The function supports various window types, including boxcar, Hamming, Hann, and Povey.

### Key Implementation Details

*   Supports periodic and symmetric window options.
*   Window length adaptation based on whether `frame_length` is provided.
*   Automatic zero-padding of the window when `frame_length` is smaller than the window size.

### Notable Dependencies and Requirements

*   `np.ndarray`

**Key Method Signatures and Interfaces**

*   `np.hamming(length)`
*   `np.hanning(length)`
*   `np.power(np.hanning(length), 0.85)`

**External Dependencies**

*   NumPy (`np`)


key: chroma:document
string_value:
CODE:
# TODO This method does not support batching yet as we are mainly focused on inference.
def spectrogram(
    waveform: np.ndarray,
    window: np.ndarray,
    frame_length: int,
    hop_length: int,
    fft_length: Optional[int] = None,
    power: Optional[float] = 1.0,
    center: bool = True,
    pad_mode: str = "reflect",
    onesided: bool = True,
    preemphasis: Optional[float] = None,
    mel_filters: Optional[np.ndarray] = None,
    mel_floor: float = 1e-10,
    log_mel: Optional[str] = None,
    reference: float = 1.0,
    min_value: float = 1e-10,
    db_range: Optional[float] = None,
    remove_dc_offset: Optional[bool] = None,
    dtype: np.dtype = np.float32,
) -> np.ndarray:
    """
    Calculates a spectrogram over one waveform using the Short-Time Fourier Transform.

    This function can create the following kinds of spectrograms:

      - amplitude spectrogram (`power = 1.0`)
      - power spectrogram (`power = 2.0`)
      - complex-valued spectrogram (`power = None`)
      - log spectrogram (use `log_mel` argument)
      - mel spectrogram (provide `mel_filters`)
      - log-mel spectrogram (provide `mel_filters` and `log_mel`)

    How this works:

      1. The input waveform is split into frames of size `frame_length` that are partially overlapping by `frame_length
         - hop_length` samples.
      2. Each frame is multiplied by the window and placed into a buffer of size `fft_length`.
      3. The DFT is taken of each windowed frame.
      4. The results are stacked into a spectrogram.

    We make a distinction between the following "blocks" of sample data, each of which may have a different lengths:

      - The analysis frame. This is the size of the time slices that the input waveform is split into.
      - The window. Each analysis frame is multiplied by the window to avoid spectral leakage.
      - The FFT input buffer. The length of this determines how many frequency bins are in the spectrogram.

    In this implementation, the window is assumed to be zero-padded to have the same size as the analysis frame. A
    padded window can be obtained from `window_function()`. The FFT input buffer may be larger than the analysis frame,
    typically the next power of two.

    Note: This function is not optimized for speed yet. It should be mostly compatible with `librosa.stft` and
    `torchaudio.functional.transforms.Spectrogram`, although it is more flexible due to the different ways spectrograms
    can be constructed.

    Args:
        waveform (`np.ndarray` of shape `(length,)`):
            The input waveform. This must be a single real-valued, mono waveform.
        window (`np.ndarray` of shape `(frame_length,)`):
            The windowing function to apply, including zero-padding if necessary. The actual window length may be
            shorter than `frame_length`, but we're assuming the array has already been zero-padded.
        frame_length (`int`):
            The length of the analysis frames in samples. With librosa this is always equal to `fft_length` but we also
            allow smaller sizes.
        hop_length (`int`):
            The stride between successive analysis frames in samples.
        fft_length (`int`, *optional*):
            The size of the FFT buffer in samples. This determines how many frequency bins the spectrogram will have.
            For optimal speed, this should be a power of two. If `None`, uses `frame_length`.
        power (`float`, *optional*, defaults to 1.0):
            If 1.0, returns the amplitude spectrogram. If 2.0, returns the power spectrogram. If `None`, returns
            complex numbers.
        center (`bool`, *optional*, defaults to `True`):
            Whether to pad the waveform so that frame `t` is centered around time `t * hop_length`. If `False`, frame
            `t` will start at time `t * hop_length`.
        pad_mode (`str`, *optional*, defaults to `"reflect"`):
            Padding mode used when `center` is `True`. Possible values are: `"constant"` (pad with zeros), `"edge"`
            (pad with edge values), `"reflect"` (pads with mirrored values).
        onesided (`bool`, *optional*, defaults to `True`):
            If True, only computes the positive frequencies and returns a spectrogram containing `fft_length // 2 + 1`
            frequency bins. If False, also computes the negative frequencies and returns `fft_length` frequency bins.
        preemphasis (`float`, *optional*)
            Coefficient for a low-pass filter that applies pre-emphasis before the DFT.
        mel_filters (`np.ndarray` of shape `(num_freq_bins, num_mel_filters)`, *optional*):
            The mel filter bank. If supplied, applies a this filter bank to create a mel spectrogram.
        mel_floor (`float`, *optional*, defaults to 1e-10):
            Minimum value of mel frequency banks.
        log_mel (`str`, *optional*):
            How to convert the spectrogram to log scale. Possible options are: `None` (don't convert), `"log"` (take
            the natural logarithm) `"log10"` (take the base-10 logarithm), `"dB"` (convert to decibels). Can only be
            used when `power` is not `None`.
        reference (`float`, *optional*, defaults to 1.0):
            Sets the input spectrogram value that corresponds to 0 dB. For example, use `np.max(spectrogram)` to set
            the loudest part to 0 dB. Must be greater than zero.
        min_value (`float`, *optional*, defaults to `1e-10`):
            The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking
            `log(0)`. For a power spectrogram, the default of `1e-10` corresponds to a minimum of -100 dB. For an
            amplitude spectrogram, the value `1e-5` corresponds to -100 dB. Must be greater than zero.
        db_range (`float`, *optional*):
            Sets the maximum dynamic range in decibels. For example, if `db_range = 80`, the difference between the
            peak value and the smallest value will never be more than 80 dB. Must be greater than zero.
        remove_dc_offset (`bool`, *optional*):
            Subtract mean from waveform on each frame, applied before pre-emphasis. This should be set to `true` in
            order to get the same results as `torchaudio.compliance.kaldi.fbank` when computing mel filters.
        dtype (`np.dtype`, *optional*, defaults to `np.float32`):
            Data type of the spectrogram tensor. If `power` is None, this argument is ignored and the dtype will be
            `np.complex64`.

    Returns:
        `nd.array` containing a spectrogram of shape `(num_frequency_bins, length)` for a regular spectrogram or shape
        `(num_mel_filters, length)` for a mel spectrogram.
    """
    window_length = len(window)

    if fft_length is None:
        fft_length = frame_length

    if frame_length > fft_length:
        raise ValueError(f"frame_length ({frame_length}) may not be larger than fft_length ({fft_length})")

    if window_length != frame_length:
        raise ValueError(f"Length of the window ({window_length}) must equal frame_length ({frame_length})")

    if hop_length <= 0:
        raise ValueError("hop_length must be greater than zero")

    if waveform.ndim != 1:
        raise ValueError(f"Input waveform must have only one dimension, shape is {waveform.shape}")

    if np.iscomplexobj(waveform):
        raise ValueError("Complex-valued input waveforms are not currently supported")

    if power is None and mel_filters is not None:
        raise ValueError(
            "You have provided `mel_filters` but `power` is `None`. Mel spectrogram computation is not yet supported for complex-valued spectrogram."
            "Specify `power` to fix this issue."
        )

    # center pad the waveform
    if center:
        padding = [(int(frame_length // 2), int(frame_length // 2))]
        waveform = np.pad(waveform, padding, mode=pad_mode)

    # promote to float64, since np.fft uses float64 internally
    waveform = waveform.astype(np.float64)
    window = window.astype(np.float64)

    # split waveform into frames of frame_length size
    num_frames = int(1 + np.floor((waveform.size - frame_length) / hop_length))

    num_frequency_bins = (fft_length // 2) + 1 if onesided else fft_length
    spectrogram = np.empty((num_frames, num_frequency_bins), dtype=np.complex64)

    # rfft is faster than fft
    fft_func = np.fft.rfft if onesided else np.fft.fft
    buffer = np.zeros(fft_length)

    timestep = 0
    for frame_idx in range(num_frames):
        buffer[:frame_length] = waveform[timestep : timestep + frame_length]

        if remove_dc_offset:
            buffer[:frame_length] = buffer[:frame_length] - buffer[:frame_length].mean()

        if preemphasis is not None:
            buffer[1:frame_length] -= preemphasis * buffer[: frame_length - 1]
            buffer[0] *= 1 - preemphasis

        buffer[:frame_length] *= window

        spectrogram[frame_idx] = fft_func(buffer)
        timestep += hop_length

    # note: ** is much faster than np.power
    if power is not None:
        spectrogram = np.abs(spectrogram, dtype=np.float64) ** power

    spectrogram = spectrogram.T

    if mel_filters is not None:
        spectrogram = np.maximum(mel_floor, np.dot(mel_filters.T, spectrogram))

    if power is not None and log_mel is not None:
        if log_mel == "log":
            spectrogram = np.log(spectrogram)
        elif log_mel == "log10":
            spectrogram = np.log10(spectrogram)
        elif log_mel == "dB":
            if power == 1.0:
                spectrogram = amplitude_to_db(spectrogram, reference, min_value, db_range)
            elif power == 2.0:
                spectrogram = power_to_db(spectrogram, reference, min_value, db_range)
            else:
                raise ValueError(f"Cannot use log_mel option '{log_mel}' with power {power}")
        else:
            raise ValueError(f"Unknown log_mel option: {log_mel}")

        spectrogram = np.asarray(spectrogram, dtype)



SUMMARY:
This is the implementation of the Short-Time Fourier Transform (STFT) in PyTorch, but with more emphasis on performance and customization options. Here's a brief explanation of the code:

1. **Preprocessing**:
	* The input waveform is centered-padded to make it at least as long as the window size.
	* If `remove_dc_offset` is True, the initial frame of the buffer is subtracted from itself to remove any DC offset.
	* If `preemphasis` is non-zero, a simple pre-emphasis filter is applied to the buffer.
2. **STFT**:
	* The STFT calculation is performed using NumPy's `rfft` function (or `fft` for one-sided STFT).
	* The result is stored in the `spectrogram` array, which has shape `(num_frames, num_frequency_bins)`, where `num_frames` is the number of frames processed so far and `num_frequency_bins` is the frequency resolution.
3. **Data Type and Shape**:
	* The spectrogram is stored as a NumPy complex array (`np.complex64`) with shape `(num_frames, num_frequency_bins)`.

However, upon closer inspection, I notice that there are some inconsistencies and potential issues:

1. **Shape assumptions**: The code assumes the `window` size is equal to the frame length (`window_length == frame_length`). However, the docstring suggests that `frame_length` might be larger than `fft_length`.
2. **Data type inconsistencies**: The spectrogram array has shape `(num_frames, num_frequency_bins)`, but it's initialized with a data type of `np.complex64`. If `power` is None and `mel_filters` is not None, the code attempts to use a data type that is inconsistent with the rest of the operation.
3. **Missing documentation**: The docstring mentions various options (e.g., `center`, `pad_mode`, `remove_dc_offset`, `preemphasis`, etc.), but does not provide sufficient information about their behavior or usage.

To improve the code, I would recommend:

1. Clarify the shape assumptions and data type consistency.
2. Provide more detailed documentation for each option.
3. Consider using a constant for the frame length to avoid magic numbers.

Here's an example of how you could refactor the code to address these issues:
```python
import numpy as np

class STFT:
    def __init__(self, frame_length, fft_len=None):
        self.frame_length = frame_length
        if fft_len is None:
            fft_len = frame_length
        self.fft_len = fft_len
        self.window_len = frame_length

    def stft(self, waveform, window, center=True, pad_mode="constant", remove_dc_offset=False, preemphasis=0):
        # ... (rest of the implementation remains similar)

    def output_shape(self):
        return (self.frame_length, self.fft_len / 2 + 1) if self.fft_len % 2 == 0 else (self.frame_length, (self.fft_len - 1) // 2 + 1)
```
In this refactored version:

* The frame length is stored as an instance variable (`frame_length`).
* The `fft_len` and `window_len` are used consistently throughout the class.
* The output shape is calculated using a separate method (`output_shape`) to ensure data type and shape consistency.


key: chroma:document
string_value:
CODE:

def spectrogram_batch(
    waveform_list: List[np.ndarray],
    window: np.ndarray,
    frame_length: int,
    hop_length: int,
    fft_length: Optional[int] = None,
    power: Optional[float] = 1.0,
    center: bool = True,
    pad_mode: str = "reflect",
    onesided: bool = True,
    preemphasis: Optional[float] = None,
    mel_filters: Optional[np.ndarray] = None,
    mel_floor: float = 1e-10,
    log_mel: Optional[str] = None,
    reference: float = 1.0,
    min_value: float = 1e-10,
    db_range: Optional[float] = None,
    remove_dc_offset: Optional[bool] = None,
    dtype: np.dtype = np.float32,
) -> List[np.ndarray]:
    """
    Calculates spectrograms for a list of waveforms using the Short-Time Fourier Transform, optimized for batch processing.
    This function extends the capabilities of the `spectrogram` function to handle multiple waveforms efficiently by leveraging broadcasting.

    It supports generating various types of spectrograms:

        - amplitude spectrogram (`power = 1.0`)
        - power spectrogram (`power = 2.0`)
        - complex-valued spectrogram (`power = None`)
        - log spectrogram (use `log_mel` argument)
        - mel spectrogram (provide `mel_filters`)
        - log-mel spectrogram (provide `mel_filters` and `log_mel`)

    How this works:

        1. The input waveform is split into frames of size `frame_length` that are partially overlapping by `frame_length
            - hop_length` samples.
        2. Each frame is multiplied by the window and placed into a buffer of size `fft_length`.
        3. The DFT is taken of each windowed frame.
        4. The results are stacked into a spectrogram.

    We make a distinction between the following "blocks" of sample data, each of which may have a different lengths:

      - The analysis frame. This is the size of the time slices that the input waveform is split into.
      - The window. Each analysis frame is multiplied by the window to avoid spectral leakage.
      - The FFT input buffer. The length of this determines how many frequency bins are in the spectrogram.

    In this implementation, the window is assumed to be zero-padded to have the same size as the analysis frame. A
    padded window can be obtained from `window_function()`. The FFT input buffer may be larger than the analysis frame,
    typically the next power of two.

    Note: This function is designed for efficient batch processing of multiple waveforms but retains compatibility with individual waveform processing methods like `librosa.stft`.

    Args:
        waveform_list (`List[np.ndarray]` with arrays of shape `(length,)`):
            The list of input waveforms, each a single-channel (mono) signal.
        window (`np.ndarray` of shape `(frame_length,)`):
            The windowing function to apply, including zero-padding if necessary.
        frame_length (`int`):
            The length of each frame for analysis.
        hop_length (`int`):
            The step size between successive frames.
        fft_length (`int`, *optional*):
            The size of the FFT buffer, defining frequency bin resolution.
        power (`float`, *optional*, defaults to 1.0):
            Determines the type of spectrogram: 1.0 for amplitude, 2.0 for power, None for complex.
        center (`bool`, *optional*, defaults to `True`):
            Whether to center-pad the waveform frames.
        pad_mode (`str`, *optional*, defaults to `"reflect"`):
            The padding strategy when `center` is `True`.
        onesided (`bool`, *optional*, defaults to `True`):
            If True, returns a one-sided spectrogram for real input signals.
        preemphasis (`float`, *optional*):
            Applies a pre-emphasis filter to each frame.
        mel_filters (`np.ndarray`, *optional*):
            Mel filter bank for converting to mel spectrogram.
        mel_floor (`float`, *optional*, defaults to 1e-10):
            Floor value for mel spectrogram to avoid log(0).
        log_mel (`str`, *optional*):
            Specifies log scaling strategy; options are None, "log", "log10", "dB".
        reference (`float`, *optional*, defaults to 1.0):
            Reference value for dB conversion in log_mel.
        min_value (`float`, *optional*, defaults to 1e-10):
            Minimum floor value for log scale conversions.
        db_range (`float`, *optional*):
            Dynamic range for dB scale spectrograms.
        remove_dc_offset (`bool`, *optional*):
            Whether to remove the DC offset from each frame.
        dtype (`np.dtype`, *optional*, defaults to `np.float32`):
            Data type of the output spectrogram.

    Returns:
        List[`np.ndarray`]: A list of spectrogram arrays, one for each input waveform.
    """
    window_length = len(window)

    if fft_length is None:
        fft_length = frame_length

    if frame_length > fft_length:
        raise ValueError(f"frame_length ({frame_length}) may not be larger than fft_length ({fft_length})")

    if window_length != frame_length:
        raise ValueError(f"Length of the window ({window_length}) must equal frame_length ({frame_length})")

    if hop_length <= 0:
        raise ValueError("hop_length must be greater than zero")

    # Check the dimensions of the waveform , and if waveform is complex
    for waveform in waveform_list:
        if waveform.ndim != 1:
            raise ValueError(f"Input waveform must have only one dimension, shape is {waveform.shape}")
        if np.iscomplexobj(waveform):
            raise ValueError("Complex-valued input waveforms are not currently supported")
    # Center pad the waveform
    if center:
        padding = [(int(frame_length // 2), int(frame_length // 2))]
        waveform_list = [
            np.pad(
                waveform,
                padding,
                mode=pad_mode,
            )
            for waveform in waveform_list
        ]
    original_waveform_lengths = [
        len(waveform) for waveform in waveform_list
    ]  # these lengths will be used to remove padding later

    # Batch pad the waveform
    max_length = max(original_waveform_lengths)
    padded_waveform_batch = np.array(
        [
            np.pad(waveform, (0, max_length - len(waveform)), mode="constant", constant_values=0)
            for waveform in waveform_list
        ],
        dtype=dtype,
    )

    # Promote to float64, since np.fft uses float64 internally
    padded_waveform_batch = padded_waveform_batch.astype(np.float64)
    window = window.astype(np.float64)

    # Split waveform into frames of frame_length size
    num_frames = int(1 + np.floor((padded_waveform_batch.shape[1] - frame_length) / hop_length))
    # these lengths will be used to remove padding later
    true_num_frames = [int(1 + np.floor((length - frame_length) / hop_length)) for length in original_waveform_lengths]
    num_batches = padded_waveform_batch.shape[0]

    num_frequency_bins = (fft_length // 2) + 1 if onesided else fft_length
    spectrogram = np.empty((num_batches, num_frames, num_frequency_bins), dtype=np.complex64)

    # rfft is faster than fft
    fft_func = np.fft.rfft if onesided else np.fft.fft
    buffer = np.zeros((num_batches, fft_length))

    for frame_idx in range(num_frames):
        timestep = frame_idx * hop_length
        buffer[:, :frame_length] = padded_waveform_batch[:, timestep : timestep + frame_length]

        if remove_dc_offset:
            buffer[:, :frame_length] -= buffer[:, :frame_length].mean(axis=1, keepdims=True)

        if preemphasis is not None:
            buffer[:, 1:frame_length] -= preemphasis * buffer[:, : frame_length - 1]
            buffer[:, 0] *= 1 - preemphasis

        buffer[:, :frame_length] *= window

        spectrogram[:, frame_idx] = fft_func(buffer)

    # Note: ** is much faster than np.power
    if power is not None:
        spectrogram = np.abs(spectrogram, dtype=np.float64) ** power

    # Apply mel filters if provided
    if mel_filters is not None:
        result = np.tensordot(spectrogram, mel_filters.T, axes=([2], [1]))
        spectrogram = np.maximum(mel_floor, result)

    # Convert to log scale if specified
    if power is not None and log_mel is not None:
        if log_mel == "log":
            spectrogram = np.log(spectrogram)
        elif log_mel == "log10":
            spectrogram = np.log10(spectrogram)
        elif log_mel == "dB":
            if power == 1.0:
                spectrogram = amplitude_to_db_batch(spectrogram, reference, min_value, db_range)
            elif power == 2.0:
                spectrogram = power_to_db_batch(spectrogram, reference, min_value, db_range)
            else:
                raise ValueError(f"Cannot use log_mel option '{log_mel}' with power {power}")
        else:
            raise ValueError(f"Unknown log_mel option: {log_mel}")

        spectrogram = np.asarray(spectrogram, dtype)

    spectrogram_list = [spectrogram[i, : true_num_frames[i], :].T for i in range(len(true_num_frames))]



SUMMARY:
**Spectral Processing Library Function**

The provided code is part of a spectral processing library that generates spectrograms from audio waveforms. It takes an input waveform batch and various parameters (such as frame length, hop size, power, etc.) to produce a batch of spectrogram arrays.

**Key Features:**

*   Supports complex-valued audio data
*   Zero-padding of the input waveform to meet requirements for FFT/IRFT procedures
*   Supports center-padded waveforms and dynamic padding based on original waveform lengths
*   Implements efficient FFT (Fast Fourier Transform) procedures using `np.fft.rfft` or `np.fft.fft`
*   Optionally applies power normalization, mel filters, log scaling with specific implementations for logarithms other than 2.0.

**Return Values:**

The function returns a batch of spectrogram arrays by flattening the time axis after processing and reshaping them into three-dimensional arrays with shape `(num_batches, num_frames, num_frequency_bins)`. The spectrograms can be viewed as complex-valued arrays that represent the power distribution across different frequencies at each frame in the input audio signal.


key: chroma:document
string_value:
CODE:

def power_to_db(
    spectrogram: np.ndarray,
    reference: float = 1.0,
    min_value: float = 1e-10,
    db_range: Optional[float] = None,
) -> np.ndarray:
    """
    Converts a power spectrogram to the decibel scale. This computes `10 * log10(spectrogram / reference)`, using basic
    logarithm properties for numerical stability.

    The motivation behind applying the log function on the (mel) spectrogram is that humans do not hear loudness on a
    linear scale. Generally to double the perceived volume of a sound we need to put 8 times as much energy into it.
    This means that large variations in energy may not sound all that different if the sound is loud to begin with.
    This compression operation makes the (mel) spectrogram features match more closely what humans actually hear.

    Based on the implementation of `librosa.power_to_db`.

    Args:
        spectrogram (`np.ndarray`):
            The input power (mel) spectrogram. Note that a power spectrogram has the amplitudes squared!
        reference (`float`, *optional*, defaults to 1.0):
            Sets the input spectrogram value that corresponds to 0 dB. For example, use `np.max(spectrogram)` to set
            the loudest part to 0 dB. Must be greater than zero.
        min_value (`float`, *optional*, defaults to `1e-10`):
            The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking
            `log(0)`. The default of `1e-10` corresponds to a minimum of -100 dB. Must be greater than zero.
        db_range (`float`, *optional*):
            Sets the maximum dynamic range in decibels. For example, if `db_range = 80`, the difference between the
            peak value and the smallest value will never be more than 80 dB. Must be greater than zero.

    Returns:
        `np.ndarray`: the spectrogram in decibels
    """
    if reference <= 0.0:
        raise ValueError("reference must be greater than zero")
    if min_value <= 0.0:
        raise ValueError("min_value must be greater than zero")

    reference = max(min_value, reference)

    spectrogram = np.clip(spectrogram, a_min=min_value, a_max=None)
    spectrogram = 10.0 * (np.log10(spectrogram) - np.log10(reference))

    if db_range is not None:
        if db_range <= 0.0:
            raise ValueError("db_range must be greater than zero")
        spectrogram = np.clip(spectrogram, a_min=spectrogram.max() - db_range, a_max=None)



SUMMARY:
**Summary**

The `power_to_db` function converts a power spectrogram to the decibel scale by applying the formula `10 * log10(spectrogram / reference)`.

**Key Implementation Details:**

* Uses numerical stability logarithm properties.
* Applies clipping to avoid `log(0)` and limits dynamic range with `db_range`.
* Computed using:
	+ `np.log10(spectrogram)`
	+ `np.log10(reference)`
	+ Multiplication by 10
	+ Subtraction of log values

**Notable Dependencies/Requirements:**

* `librosa` (based on the implementation)
* `numpy` (`np.ndarray`, `np.clip`, `np.log10`)
* `Optional` (for `db_range` parameter)

**Primary Purpose:** Convert power spectrogram to decibel scale.


key: chroma:document
string_value:
CODE:

def power_to_db_batch(
    spectrogram: np.ndarray,
    reference: float = 1.0,
    min_value: float = 1e-10,
    db_range: Optional[float] = None,
) -> np.ndarray:
    """
    Converts a batch of power spectrograms to the decibel scale. This computes `10 * log10(spectrogram / reference)`,
    using basic logarithm properties for numerical stability.

    This function supports batch processing, where each item in the batch is an individual power (mel) spectrogram.

    Args:
        spectrogram (`np.ndarray`):
            The input batch of power (mel) spectrograms. Expected shape is (batch_size, *spectrogram_shape).
            Note that a power spectrogram has the amplitudes squared!
        reference (`float`, *optional*, defaults to 1.0):
            Sets the input spectrogram value that corresponds to 0 dB. For example, use `np.max(spectrogram)` to set
            the loudest part to 0 dB. Must be greater than zero.
        min_value (`float`, *optional*, defaults to `1e-10`):
            The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking
            `log(0)`. The default of `1e-10` corresponds to a minimum of -100 dB. Must be greater than zero.
        db_range (`float`, *optional*):
            Sets the maximum dynamic range in decibels. For example, if `db_range = 80`, the difference between the
            peak value and the smallest value will never be more than 80 dB. Must be greater than zero.

    Returns:
        `np.ndarray`: the batch of spectrograms in decibels
    """
    if reference <= 0.0:
        raise ValueError("reference must be greater than zero")
    if min_value <= 0.0:
        raise ValueError("min_value must be greater than zero")

    reference = max(min_value, reference)

    spectrogram = np.clip(spectrogram, a_min=min_value, a_max=None)
    spectrogram = 10.0 * (np.log10(spectrogram) - np.log10(reference))

    if db_range is not None:
        if db_range <= 0.0:
            raise ValueError("db_range must be greater than zero")
        # Apply db_range clipping per batch item
        max_values = spectrogram.max(axis=(1, 2), keepdims=True)
        spectrogram = np.clip(spectrogram, a_min=max_values - db_range, a_max=None)



SUMMARY:
**Summary**

The `power_to_db_batch` function converts a batch of power spectrograms to decibel scale. It takes in a batch of power (mel) spectrograms, reference value, minimum output value, and optional DB range parameter.

**Main Purpose:**
Converts power spectrograms to decibels using the formula 10 * log10(spectrogram / reference).

**Key Implementation Details:**

* `np.log10` is used for numerical stability.
* Clipping applied to minimize `log(0)` issues.
* Optional DB range clipping per batch item.

**Notable Dependencies/Requirements:**

* `numpy` library for numerical computations.
* `Optional` type hint for `db_range` parameter.
* `ValueError` raises for invalid reference and min_value inputs.


key: chroma:document
string_value:
CODE:

def amplitude_to_db(
    spectrogram: np.ndarray,
    reference: float = 1.0,
    min_value: float = 1e-5,
    db_range: Optional[float] = None,
) -> np.ndarray:
    """
    Converts an amplitude spectrogram to the decibel scale. This computes `20 * log10(spectrogram / reference)`, using
    basic logarithm properties for numerical stability.

    The motivation behind applying the log function on the (mel) spectrogram is that humans do not hear loudness on a
    linear scale. Generally to double the perceived volume of a sound we need to put 8 times as much energy into it.
    This means that large variations in energy may not sound all that different if the sound is loud to begin with.
    This compression operation makes the (mel) spectrogram features match more closely what humans actually hear.

    Args:
        spectrogram (`np.ndarray`):
            The input amplitude (mel) spectrogram.
        reference (`float`, *optional*, defaults to 1.0):
            Sets the input spectrogram value that corresponds to 0 dB. For example, use `np.max(spectrogram)` to set
            the loudest part to 0 dB. Must be greater than zero.
        min_value (`float`, *optional*, defaults to `1e-5`):
            The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking
            `log(0)`. The default of `1e-5` corresponds to a minimum of -100 dB. Must be greater than zero.
        db_range (`float`, *optional*):
            Sets the maximum dynamic range in decibels. For example, if `db_range = 80`, the difference between the
            peak value and the smallest value will never be more than 80 dB. Must be greater than zero.

    Returns:
        `np.ndarray`: the spectrogram in decibels
    """
    if reference <= 0.0:
        raise ValueError("reference must be greater than zero")
    if min_value <= 0.0:
        raise ValueError("min_value must be greater than zero")

    reference = max(min_value, reference)

    spectrogram = np.clip(spectrogram, a_min=min_value, a_max=None)
    spectrogram = 20.0 * (np.log10(spectrogram) - np.log10(reference))

    if db_range is not None:
        if db_range <= 0.0:
            raise ValueError("db_range must be greater than zero")
        spectrogram = np.clip(spectrogram, a_min=spectrogram.max() - db_range, a_max=None)



SUMMARY:
**Summary**

This function converts an amplitude spectrogram to the decibel scale using the formula `20 * log10(spectrogram / reference)`, with support for clipping and dynamic range limits.

**Main Purpose and Functionality**

*Converts amplitude spectrogram to decibels*

**Implementation Details**

* Computes `20 * log10(spectrogram / reference)`
* Clips values to minimum and maximum bounds:
	+ `np.clip(spectrogram, a_min=min_value, a_max=None)` (avoiding `log(0)`)
	+ `np.clip(spectrogram, a_min=spectrogram.max() - db_range, a_max=None)` with dynamic range limit (`db_range`)

**Notable Dependencies/Requirements**

* `np.ndarray` for input spectrogram
* `Optional[float]` for minimum value and dynamic range limit (`db_range`)
* Referenced functions:
	+ `np.log10`
	+ `np.max`

**Interface:**

* Input: `spectrogram`, `reference`, `min_value`, `db_range`
* Output: `np.ndarray` (spectrogram in decibels)


key: chroma:document
string_value:
CODE:

def amplitude_to_db_batch(
    spectrogram: np.ndarray, reference: float = 1.0, min_value: float = 1e-5, db_range: Optional[float] = None
) -> np.ndarray:
    """
    Converts a batch of amplitude spectrograms to the decibel scale. This computes `20 * log10(spectrogram / reference)`,
    using basic logarithm properties for numerical stability.

    The function supports batch processing, where each item in the batch is an individual amplitude (mel) spectrogram.

    Args:
        spectrogram (`np.ndarray`):
            The input batch of amplitude (mel) spectrograms. Expected shape is (batch_size, *spectrogram_shape).
        reference (`float`, *optional*, defaults to 1.0):
            Sets the input spectrogram value that corresponds to 0 dB. For example, use `np.max(spectrogram)` to set
            the loudest part to 0 dB. Must be greater than zero.
        min_value (`float`, *optional*, defaults to `1e-5`):
            The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking
            `log(0)`. The default of `1e-5` corresponds to a minimum of -100 dB. Must be greater than zero.
        db_range (`float`, *optional*):
            Sets the maximum dynamic range in decibels. For example, if `db_range = 80`, the difference between the
            peak value and the smallest value will never be more than 80 dB. Must be greater than zero.

    Returns:
        `np.ndarray`: the batch of spectrograms in decibels
    """
    if reference <= 0.0:
        raise ValueError("reference must be greater than zero")
    if min_value <= 0.0:
        raise ValueError("min_value must be greater than zero")

    reference = max(min_value, reference)

    spectrogram = np.clip(spectrogram, a_min=min_value, a_max=None)
    spectrogram = 20.0 * (np.log10(spectrogram) - np.log10(reference))

    if db_range is not None:
        if db_range <= 0.0:
            raise ValueError("db_range must be greater than zero")
        # Apply db_range clipping per batch item
        max_values = spectrogram.max(axis=(1, 2), keepdims=True)
        spectrogram = np.clip(spectrogram, a_min=max_values - db_range, a_max=None)



SUMMARY:
Here is a concise summary of the provided function:

**Function Name:** `amplitude_to_db_batch`

**Main Purpose:** Converts a batch of amplitude spectrograms to decibel scale.

**Key Implementation Details:**

* Computes `20 * log10(spectrogram / reference)` for each spectrogram in batch.
* Clips values to prevent taking `log(0)` at minimum value (`min_value`).
* Applicably clamps the dynamic range to `db_range` if specified.

**Notable Digits and Dependencies:**

* Input shape: (batch_size, *spectrogram_shape)
* Reference value must be greater than zero.
* Minimum value constraint ensures against taking log(0) for numerical stability.
* Optional parameter `db_range` allows custom dynamic range clipping.
* External dependencies:
	+ NumPy (`np`) for array operations.


key: chroma:document
string_value:
CODE:

def get_mel_filter_banks(
    nb_frequency_bins: int,
    nb_mel_filters: int,
    frequency_min: float,
    frequency_max: float,
    sample_rate: int,
    norm: Optional[str] = None,
    mel_scale: str = "htk",
) -> np.array:
    warnings.warn(
        "The function `get_mel_filter_banks` is deprecated and will be removed in version 4.31.0 of Transformers",
        FutureWarning,
    )
    return mel_filter_bank(
        num_frequency_bins=nb_frequency_bins,
        num_mel_filters=nb_mel_filters,
        min_frequency=frequency_min,
        max_frequency=frequency_max,
        sampling_rate=sample_rate,
        norm=norm,
        mel_scale=mel_scale,


SUMMARY:
**Summary**

`get_mel_filter_banks` calculates Mel filter banks for audio processing. It takes parameters such as frequency bins, mel filters, and sampling rate to generate a matrix of Mel coefficients. The function uses `mel_filter_bank` internal implementation.

**Key Details**

* Parameters:
	+ `num_frequency_bins`: int (number of frequency bins)
	+ `num_mel_filters`: int (number of mel filters)
	+ `min_frequency`, `max_frequency`: float (frequency limits)
	+ `sampling_rate`: int (audio sampling rate)
	+ `norm`: str, optional (normalization type, default: None)
	+ `mel_scale`: str (Mel scale, default: "htk")
* Returns: np.array (Mel filter bank matrix)
* Dependencies:
	+ `mel_filter_bank` internal implementation
	+ `Transformers` library version 4.31.0 and later ( deprecated warning)

**Context**: This function is marked as deprecated, indicating that it will be removed in a future version of the Transformers library.


key: chroma:document
string_value:
CODE:

def fram_wave(waveform: np.array, hop_length: int = 160, fft_window_size: int = 400, center: bool = True):
    """
    In order to compute the short time fourier transform, the waveform needs to be split in overlapping windowed
    segments called `frames`.

    The window length (window_length) defines how much of the signal is contained in each frame, while the hop length
    defines the step between the beginning of each new frame.


    Args:
        waveform (`np.array` of shape `(sample_length,)`):
            The raw waveform which will be split into smaller chunks.
        hop_length (`int`, *optional*, defaults to 160):
            Step between each window of the waveform.
        fft_window_size (`int`, *optional*, defaults to 400):
            Defines the size of the window.
        center (`bool`, defaults to `True`):
            Whether or not to center each frame around the middle of the frame. Centering is done by reflecting the
            waveform on the left and on the right.

    Return:
        framed_waveform (`np.array` of shape `(waveform.shape // hop_length , fft_window_size)`):
            The framed waveforms that can be fed to `np.fft`.
    """
    warnings.warn(
        "The function `fram_wave` is deprecated and will be removed in version 4.31.0 of Transformers",
        FutureWarning,
    )
    frames = []
    for i in range(0, waveform.shape[0] + 1, hop_length):
        if center:
            half_window = (fft_window_size - 1) // 2 + 1
            start = i - half_window if i > half_window else 0
            end = i + half_window if i < waveform.shape[0] - half_window else waveform.shape[0]
            frame = waveform[start:end]
            if start == 0:
                padd_width = (-i + half_window, 0)
                frame = np.pad(frame, pad_width=padd_width, mode="reflect")

            elif end == waveform.shape[0]:
                padd_width = (0, (i - waveform.shape[0] + half_window))
                frame = np.pad(frame, pad_width=padd_width, mode="reflect")

        else:
            frame = waveform[i : i + fft_window_size]
            frame_width = frame.shape[0]
            if frame_width < waveform.shape[0]:
                frame = np.lib.pad(
                    frame, pad_width=(0, fft_window_size - frame_width), mode="constant", constant_values=0
                )
        frames.append(frame)

    frames = np.stack(frames, 0)


SUMMARY:
**Summary: fram_wave Function**

This function computes the short-time Fourier transform (STFT) of a given waveform by splitting it into overlapping windowed segments called "frames".

**Primary Purpose:** Computs STFT of a waveform.

**Key Implementation Details:**

* Splits waveform into frames with specified hop length and FFT window size.
* Frames are centered around the middle of each frame using reflection.
* Function can be used to extract features for audio processing or analysis tasks.

**Notable Dependencies:**

* `np` (NumPy) library for numerical computations.
* `np.fft` module for Fourier transform calculations.


key: chroma:document
string_value:
CODE:

def stft(frames: np.array, windowing_function: np.array, fft_window_size: int = None):
    """
    Calculates the complex Short-Time Fourier Transform (STFT) of the given framed signal. Should give the same results
    as `torch.stft`.

    Args:
        frames (`np.array` of dimension `(num_frames, fft_window_size)`):
            A framed audio signal obtained using `audio_utils.fram_wav`.
        windowing_function (`np.array` of dimension `(nb_frequency_bins, nb_mel_filters)`:
            A array representing the function that will be used to reduces the amplitude of the discontinuities at the
            boundaries of each frame when computing the STFT. Each frame will be multiplied by the windowing_function.
            For more information on the discontinuities, called *Spectral leakage*, refer to [this
            tutorial]https://download.ni.com/evaluation/pxi/Understanding%20FFTs%20and%20Windowing.pdf
        fft_window_size (`int`, *optional*):
            Size of the window om which the Fourier transform is applied. This controls the frequency resolution of the
            spectrogram. 400 means that the fourrier transform is computed on windows of 400 samples. The number of
            frequency bins (`nb_frequency_bins`) used to divide the window into equal strips is equal to
            `(1+fft_window_size)//2`. An increase of the fft_window_size slows the calculus time proportionnally.

    Example:

    ```python
    >>> from transformers.audio_utils import stft, fram_wave
    >>> import numpy as np

    >>> audio = np.random.rand(50)
    >>> fft_window_size = 10
    >>> hop_length = 2
    >>> framed_audio = fram_wave(audio, hop_length, fft_window_size)
    >>> spectrogram = stft(framed_audio, np.hanning(fft_window_size + 1))
    ```

    Returns:
        spectrogram (`np.ndarray`):
            A spectrogram of shape `(num_frames, nb_frequency_bins)` obtained using the STFT algorithm
    """
    warnings.warn(
        "The function `stft` is deprecated and will be removed in version 4.31.0 of Transformers",
        FutureWarning,
    )
    frame_size = frames.shape[1]

    if fft_window_size is None:
        fft_window_size = frame_size

    if fft_window_size < frame_size:
        raise ValueError("FFT size must greater or equal the frame size")
    # number of FFT bins to store
    nb_frequency_bins = (fft_window_size >> 1) + 1

    spectrogram = np.empty((len(frames), nb_frequency_bins), dtype=np.complex64)
    fft_signal = np.zeros(fft_window_size)

    for f, frame in enumerate(frames):
        if windowing_function is not None:
            np.multiply(frame, windowing_function, out=fft_signal[:frame_size])
        else:
            fft_signal[:frame_size] = frame
        spectrogram[f] = np.fft.fft(fft_signal, axis=0)[:nb_frequency_bins]


SUMMARY:
**Summary: Time Domain Short-TFT Function**

Calculates the complex Short-Time Fourier Transform (STFT) of input frames using FFT.

**Functionality:**

1. Applies windowing function to each frame for spectral leakage reduction
2. Computes STFT on windowed frames, storing result in a spectrogram array

**Code Focus:**

* `np.fft.fft` for frequency transform
* Windowing function application and spectral leakage reduction
* Iterative computation of individual spectrogram lines

**Input Dependencies:**

* `frames`: input audio signal framed with specific hop length and fft window size
* `windowing_function`: optional, reduces amplitude of discontinuities at frame boundaries (`np.hanning` used as an example)

**Output:**
A 2D spectrogram array of shape `(num_frames, nb_frequency_bins)`, representing the complex STFT output.

Note: The function is deprecated and will be removed in version 4.31.0 of Transformers.


key: chroma:document
string_value:
CODE:

def get_dataloader_sampler(dataloader):
    if hasattr(dataloader, "batch_sampler") and dataloader.batch_sampler is not None:
        return get_dataloader_sampler(dataloader.batch_sampler)
    elif hasattr(dataloader, "sampler"):


SUMMARY:
**Summary:**

The `get_dataloader_sampler` function determines the sampler for a PyTorch DataLoader. It recursively checks for a `batch_sampler` attribute and if present, returns its connected sampler. If not, it directly returns the `sampler` of the DataLoader.

**Key Implementation Details:**

* Recursively uses `batch_sampler` to find a nested sampler if present.
* Directly returns `sampler` when no `batch_sampler` is found.
* Both `batch_sampler` and `sampler` are key attributes of PyTorch DataLoaders.

No notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

def atleast_1d(tensor_or_array: Union[torch.Tensor, np.ndarray]):
    if isinstance(tensor_or_array, torch.Tensor):
        if hasattr(torch, "atleast_1d"):
            tensor_or_array = torch.atleast_1d(tensor_or_array)
        elif tensor_or_array.ndim < 1:
            tensor_or_array = tensor_or_array[None]
    else:
        tensor_or_array = np.atleast_1d(tensor_or_array)


SUMMARY:
**Summary: `atleast_1d` Function**

**Purpose:** Upscales a 0D or lower-dimensional tensor/array to at least 1D.

**Primary Purpose:**
Ensures input is a 1D tensor or array for further processing.

**Key Implementation Details:**

* Checks if input `tensor_or_array` is a PyTorch tensor (<code>`torch.Tensor</code>).
* If PyTorch, attempts to use built-in `atleast_1d` function (<code>`torch.atleast_1d`).
* If not, adds singleton dimension (`None`) if necessary.
* If input is NumPy array, uses same scaling logic as PyTorch.

**Notable Dependencies/Requirements:**

*: Both `torch` and `numpy` libraries are required for this function to work.


key: chroma:document
string_value:
CODE:

def torch_pad_and_concatenate(tensor1, tensor2, padding_index=-100):
    """Concatenates `tensor1` and `tensor2` on first axis, applying padding on the second if necessary."""
    tensor1 = atleast_1d(tensor1)
    tensor2 = atleast_1d(tensor2)

    if len(tensor1.shape) == 1 or tensor1.shape[1] == tensor2.shape[1]:
        return torch.cat((tensor1, tensor2), dim=0)

    # Let's figure out the new shape
    new_shape = (tensor1.shape[0] + tensor2.shape[0], max(tensor1.shape[1], tensor2.shape[1])) + tensor1.shape[2:]

    # Now let's fill the result tensor
    result = tensor1.new_full(new_shape, padding_index)
    result[: tensor1.shape[0], : tensor1.shape[1]] = tensor1
    result[tensor1.shape[0] :, : tensor2.shape[1]] = tensor2


SUMMARY:
**Summary:**

### Purpose and Functionality:

The `torch_pad_and_concatenate` function concatenates two tensors (`tensor1` and `tensor2`) on the first axis, applying padding on the second axis if necessary.

### Key Implementation Details:

* Uses `torch.cat` to concatenate tensors along the first axis.
* Calculates new shape based on tensor dimensions.
* Creates a result tensor with new shape using `tensor.new_full`.
* Fills result tensor with elements from `tensor1` and `tensor2`.

### Notable Dependencies or Requirements:

* `atleast_1d`: ensures input tensors have at least one dimension.
* `torch.cat`: concatenates tensors along the specified axis.
* `torch.tensor`: creates a new PyTorch tensor.

**RAG Entry (searchable and contextual):**

`torch_pad_and_concatenate(tensor1, tensor2, padding_index=-100)`
`Concatenates tensors on first axis, applying padding on second if necessary.`


key: chroma:document
string_value:
CODE:

def numpy_pad_and_concatenate(array1, array2, padding_index=-100):
    """Concatenates `array1` and `array2` on first axis, applying padding on the second if necessary."""
    array1 = atleast_1d(array1)
    array2 = atleast_1d(array2)

    if len(array1.shape) == 1 or array1.shape[1] == array2.shape[1]:
        return np.concatenate((array1, array2), axis=0)

    # Let's figure out the new shape
    new_shape = (array1.shape[0] + array2.shape[0], max(array1.shape[1], array2.shape[1])) + array1.shape[2:]

    # Now let's fill the result tensor
    result = np.full_like(array1, padding_index, shape=new_shape)
    result[: array1.shape[0], : array1.shape[1]] = array1
    result[array1.shape[0] :, : array2.shape[1]] = array2


SUMMARY:
Here is a concise summary of the provided code chunk:

**Function Summary: `numpy_pad_and_concatenate`**

**Purpose:** Concatenates two 4D arrays (`array1` and `array2`) on the first axis, padding the second array if necessary.

**Key Implementation Details:**

* Verifies input dimensions and pads `array2` to match `array1`'s shape.
* Fills a new tensor with padded values using NumPy's full_like function.
* Copies data from `array1` and `array2` into the resulting tensor, ensuring alignment along axis 1.

**Notable Dependencies:**

* Numpy (`np`)
* `atleast_1d` function
* `full_like` function

**Functions Called:** `np.concatenate`, `np.full_like`

This function appears to be part of a larger library or module providing NumPy-specific functionality for array manipulation.


key: chroma:document
string_value:
CODE:

def nested_concat(tensors, new_tensors, padding_index=-100):
    """
    Concat the `new_tensors` to `tensors` on the first dim and pad them on the second if needed. Works for tensors or
    nested list/tuples/dict of tensors.
    """
    if not (isinstance(tensors, torch.Tensor) and isinstance(new_tensors, torch.Tensor)):
        assert (
            type(tensors) is type(new_tensors)
        ), f"Expected `tensors` and `new_tensors` to have the same type but found {type(tensors)} and {type(new_tensors)}."
    if isinstance(tensors, (list, tuple)):
        return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
    elif isinstance(tensors, torch.Tensor):
        return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)
    elif isinstance(tensors, Mapping):
        return type(tensors)(
            {k: nested_concat(t, new_tensors[k], padding_index=padding_index) for k, t in tensors.items()}
        )
    elif isinstance(tensors, np.ndarray):
        return numpy_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)
    else:


SUMMARY:
**Function Summary**

`nested_concat` functions concatanates and pads `new_tensors` to `tensors` in a nested manner, supporting tensors, lists/tuples, dictionaries, and numpy arrays.

**Key Implementation Details**

*   Checks the types of `tensors` and `new_tensors`, asserting they must be the same.
*   Supports concatenation on first dimension (dim=0) using `torch_pad_and_concatenate`.
*   Recursively handles nested tensors on all supported data structures (`list`, tuple, dict, np.ndarray).
*   Raises a `TypeError` for unsupported data types.

**Notable Dependencies/Requirements**

*   `torch.Tensor`
*   `torch.pad_and_concatenate`
*   `type(tensors)`
*   `type(new_tensors)`
*   `zip(tensors, new_tensors)`


key: chroma:document
string_value:
CODE:

def find_batch_size(tensors):
    """
    Find the first dimension of a tensor in a nested list/tuple/dict of tensors.
    """
    if isinstance(tensors, (list, tuple)):
        for t in tensors:
            result = find_batch_size(t)
            if result is not None:
                return result
    elif isinstance(tensors, Mapping):
        for key, value in tensors.items():
            result = find_batch_size(value)
            if result is not None:
                return result
    elif isinstance(tensors, torch.Tensor):
        return tensors.shape[0] if len(tensors.shape) >= 1 else None
    elif isinstance(tensors, np.ndarray):


SUMMARY:
**Summary:**

`find_batch_size` function recursively traverses a nested data structure (list, tuple, dict, PyTorch tensor, or NumPy array) to find the first dimension's index.

**Key Implementation Details:**

- Recursively checks list and tuple elements
- Recursively checks dictionary values
- Returns tensor shape[0] if available

**Notable Dependencies/Requirements:**

- `torch` (torch.Tensor type checking)
- `numpy` (np.ndarray type checking)

This function aims to extract the batch size from a data structure, allowing for handling of different tensor representations. The implementation utilizes Python's built-in data structures and tensors' shape attributes.


key: chroma:document
string_value:
CODE:

def nested_numpify(tensors):
    "Numpify `tensors` (even if it's a nested list/tuple/dict of tensors)."
    if isinstance(tensors, (list, tuple)):
        return type(tensors)(nested_numpify(t) for t in tensors)
    if isinstance(tensors, Mapping):
        return type(tensors)({k: nested_numpify(t) for k, t in tensors.items()})

    t = tensors.cpu()
    if t.dtype == torch.bfloat16:
        # As of Numpy 1.21.4, NumPy does not support bfloat16 (see
        # https://github.com/numpy/numpy/blob/a47ecdea856986cd60eabbd53265c2ca5916ad5d/doc/source/user/basics.types.rst ).
        # Until Numpy adds bfloat16, we must convert float32.
        t = t.to(torch.float32)


SUMMARY:
**numpify Function Summary**
=====================================

### Purpose and Functionality

The `nested_numpify` function recursively converts PyTorch tensors and their nested structures into NumPy arrays, handling types including lists, tuples, dictionaries, and tensor data types.

### Key Implementation Details

*   Recursively traverses input tensors (lists, tuples, dictionaries) and conversions are performed for each element or key-value pair.
*   Tensors with `torch.bfloat16` data type are converted to `torch.float32`.
*   The resulting NumPy array is retrieved from the converted PyTorch tensor (`t.numpy()`).

### Notable Dependencies and Requirements

*   References: None
*   External Dependencies:
    *   PyTorch library
    *   NumPy library (specifically, version 1.21.4 for `bfloat16` support)
*   Implemented Interfaces:
    *   Converts tensors to NumPy arrays
    *   Handles recursive traversal of tensor structures


key: chroma:document
string_value:
CODE:

def nested_detach(tensors):
    "Detach `tensors` (even if it's a nested list/tuple/dict of tensors)."
    if isinstance(tensors, (list, tuple)):
        return type(tensors)(nested_detach(t) for t in tensors)
    elif isinstance(tensors, Mapping):
        return type(tensors)({k: nested_detach(t) for k, t in tensors.items()})


SUMMARY:
**Function Summary:**

`nested_detach`: Detaches nested tensors from gradients, handling multiple data types (lists/tuples/dicts of tensors).

**Key Details:**

* Uses recursion to handle nested structures
* Handles three data types:
	+ Lists/Tuples: recurse on individual elements using `nested_detach(t) for t in tensors`
	+ Dictionaries: use dictionary comprehension with `k: nested_detach(t)` for key-value pairs
	+ PyTorch Tensors: use `detach()` method directly
* Returns detached tensor(s)

**Notable Dependencies/Interfaces:**

* `torch.Tensor` interface
* `list`, `tuple`, and `Mapping` data types


key: chroma:document
string_value:
CODE:

def nested_xla_mesh_reduce(tensors, name):
    if is_torch_xla_available():
        import torch_xla.core.xla_model as xm

        if isinstance(tensors, (list, tuple)):
            return type(tensors)(nested_xla_mesh_reduce(t, f"{name}_{i}") for i, t in enumerate(tensors))
        if isinstance(tensors, Mapping):
            return type(tensors)(
                {k: nested_xla_mesh_reduce(t, f"{name}_{i}") for i, (k, t) in enumerate(tensors.items())}
            )

        tensors = atleast_1d(tensors)
        return xm.mesh_reduce(name, tensors, torch.cat)
    else:


SUMMARY:
**Summary:**

The `nested_xla_mesh_reduce` function reduces a nested tensor by leveraging PyTorch's XLA acceleration under the hood (`torch_xla.core.xla_model`). It recursively applies mesh reduction to tensors in various data structures (lists, tuples, and dictionaries), utilizing `torch.cat` for concatenation and `atleast_1d` for conversion to 1D tensors.

**Key Implementation Details:**

* Recursively reduces nested tensors using XLA's `mesh_reduce`
* Utilizes `torch_cat` for concatenation
* Ensures input tensors have shape compatible with mesh reduction

**Notable Dependencies/Requirements:**

* PyTorch (`torch_xla.core.xla_model`)
* `torch.cat` function


key: chroma:document
string_value:
CODE:

def distributed_concat(tensor: Any, num_total_examples: Optional[int] = None) -> Any:
    try:
        if isinstance(tensor, (tuple, list)):
            return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
        if isinstance(tensor, Mapping):
            return type(tensor)({k: distributed_concat(t, num_total_examples) for k, t in tensor.items()})
        tensor = atleast_1d(tensor).contiguous()
        output_tensors = [tensor.clone() for _ in range(dist.get_world_size())]
        dist.all_gather(output_tensors, tensor)
        concat = torch.cat(output_tensors, dim=0)

        # truncate the dummy elements added by SequentialDistributedSampler
        if num_total_examples is not None:
            concat = concat[:num_total_examples]
        return concat
    except AssertionError:


SUMMARY:
**Function Summary: Distributed Concatenation**

* **Purpose**: Concatenates tensors in a distributed manner, handling various input types (tensor, tuple, list, and mapping).
* **Implementation Details**:
	+ Checks input type and applies transformation accordingly.
	+ Splits tensor into individual ranks using `dist.get_world_size()` and collects them via `dist.all_gather`.
	+ Concatenates tensors along dimension 0 using `torch.cat`.
	+ Optionally truncates concatenated output based on `num_total_examples`.
* **Notable Dependencies**:
	+ `dist` (distributed training framework)
	+ ` atleast_1d`, `contiguous`, and `Tensor` (PyTorch tensor manipulation)
	+ `Mapping` (Python dictionary)

Note: The function raises an AssertionError if not currently using distributed training.


key: chroma:document
string_value:
CODE:

def distributed_broadcast_scalars(
    scalars: List[Union[int, float]],
    num_total_examples: Optional[int] = None,
    device: Optional[torch.device] = torch.device("cuda"),
) -> torch.Tensor:
    try:
        tensorized_scalar = torch.tensor(scalars).to(device)
        output_tensors = [tensorized_scalar.clone() for _ in range(dist.get_world_size())]
        dist.all_gather(output_tensors, tensorized_scalar)
        concat = torch.cat(output_tensors, dim=0)

        # truncate the dummy elements added by SequentialDistributedSampler
        if num_total_examples is not None:
            concat = concat[:num_total_examples]
        return concat
    except AssertionError:


SUMMARY:
**Summary: Distributed Broadcast Scalar Function**

**Purpose and Functionality:**
`distributed_broadcast_scalars`: Broadcasting scalars to all GPUs in a distributed PyTorch environment.

**Key Implementation Details:**

* Creates a tensor from input `scalars` using `torch.tensor()` and moves it to the specified device (`cuda` by default).
* Synchronizes data across all processes using `dist.all_gather()` and creates a broadcasted tensor with size equal to the total number of examples.
* Optionally truncates the output tensor based on the provided `num_total_examples`.

**Notable Dependencies and Requirements:**

* `torch`: PyTorch library for distributed training.
* `dist`: Distributed PyTorch module for parallelization.
* `torch.device`: PyTorch device manager.

**Return Value:**
A broadcasted tensor containing the input scalars.


key: chroma:document
string_value:
CODE:

def reissue_pt_warnings(caught_warnings):
    # Reissue warnings
    if len(caught_warnings) > 1:
        for w in caught_warnings:
            if w.category is not UserWarning:


SUMMARY:
**Summary:**

`reissue_pt_warnings` function reissues PTLiveTemplateWarning instances that are not `UserWarnings`. It iterates through the input list `caught_warnings`, skipping `UserWarning` instances.

**Key Implementation Details:**

* Iterates over `caught_warnings` to reissue non-`UserWarning` warnings
* Uses `warnings.warn` with warning category and message from the input warning instance

**Notable Dependencies/Requirements:**

* `warnings` module (specifically, `warnings.warn` function)
* `PTLiveTemplateWarning` class (input parameter is an instance of this class)


key: chroma:document
string_value:
CODE:

class DistributedSamplerWithLoop(DistributedSampler):
    """
    Like a torch.utils.data.distributed.DistributedSampler` but loops at the end back to the beginning of the shuffled
    samples to make each process have a round multiple of batch_size samples.

    Args:
        dataset (`torch.utils.data.Dataset`):
            Dataset used for sampling.
        batch_size (`int`):
            The batch size used with this sampler
        kwargs (`Dict[str, Any]`, *optional*):
            All other keyword arguments passed to `DistributedSampler`.
    """

    def __init__(self, dataset, batch_size, **kwargs):
        super().__init__(dataset, **kwargs)
        self.batch_size = batch_size

    def __iter__(self):
        indices = list(super().__iter__())
        remainder = 0 if len(indices) % self.batch_size == 0 else self.batch_size - len(indices) % self.batch_size
        # DistributedSampler already added samples from the beginning to make the number of samples a round multiple
        # of the world size, so we skip those.
        start_remainder = 1 if self.rank < len(self.dataset) % self.num_replicas else 0
        indices += indices[start_remainder : start_remainder + remainder]


SUMMARY:
**DistributedSamplerWithLoop Class Summary**

### Overview

A custom `DistributedSampler` class that loops at the end back to the beginning of shuffled samples, ensuring each process receives a round multiple of batch_size samples.

### Key Functionality

- Iterates over dataset with looped sampling
- Adjusts indices to make number of samples a round multiple of batch size for each process
- Skips initial batch with remaining indices for distributed sampler's "start" offset

### Notable Implementation Details

- Uses `super().__iter__()` to inherit original DistributedSampler iteration behavior
- Calculates adjusted remainder and adds it to the end of the iterator's sample indices

### Dependencies and Requirements

* `torch.utils.data.Dataset`
* `train` - `self.dataset` for dataset usage in `super().__init__()
* Inherits from `torch.utils.data.distributed.DistributedSampler`

### Notable Signature
```python
def __iter__(self):
    # ...
```
Note: `Indices` calculated and returned via iterator object.


key: chroma:document
string_value:
CODE:

class EvalLoopContainer:
    """
    Container to store intermediate results of evaluation loop

    Args:
        do_nested_concat (`bool`, *optional*, defaults to `True`):
            If set to `True`, each iteration will recursively concatenate a new object containing tensors to
            the existing stored tensors, provided that the structure of the existing object and the new one
            are identical. If set to `False`, all newly added tensors will be stored in a list.
        padding_index (`int`, *optional*, defaults to -100):
            Value used to pad tensors of different shapes when `do_nested_concat=True`.
    """

    def __init__(self, do_nested_concat: bool = True, padding_index: int = -100):
        self.do_nested_concat = do_nested_concat
        self.padding_index = padding_index
        self.tensors = None
        self.arrays = None

    def add(self, tensors) -> None:
        """Add tensors to the stored objects. If `do_nested_concat=True`, the tensors will be concatenated recursively."""
        if self.tensors is None:
            self.tensors = tensors if self.do_nested_concat else [tensors]
        elif self.do_nested_concat:
            self.tensors = nested_concat(self.tensors, tensors, padding_index=self.padding_index)
        else:
            self.tensors.append(tensors)

    def to_cpu_and_numpy(self) -> None:
        """Move tensors in stored objects to CPU and convert them to numpy arrays."""

        # Check if we have something to add, if not just return
        if self.tensors is None:
            return

        new_arrays = nested_numpify(self.tensors)
        if self.arrays is None:
            self.arrays = new_arrays
        elif self.do_nested_concat:
            self.arrays = nested_concat(self.arrays, new_arrays, padding_index=self.padding_index)
        else:
            self.arrays.extend(new_arrays)

        # reset device tensors after adding to cpu
        self.tensors = None

    def get_arrays(self):
        """Returns the numpified and moved to CPU stored objects."""
        self.to_cpu_and_numpy()


SUMMARY:
**EvalLoopContainer Class Summary**

The `EvalLoopContainer` class is designed to store intermediate results of evaluation loops. It provides methods for adding tensors, handling nested concatenation, and moving data to CPU.

**Key Features:**

* Storages tensors in either a single object or a list (optional)
* Recursively concatenates new tensors when `do_nested_concat=True`
* Moves tensors to CPU and converts them to numpy arrays (`to_cpu_and_numpy`)
* Provides a method to retrieve numpified and moved-to-CPU stored objects (`get_arrays`)

**Implementation Details:**

* Uses the `nested_concat` function for recursive concatenation and `nested_numpify` function for converting tensors to numpy arrays.
* Stores tensors in either `tensors` or `arrays` attribute, depending on the `do_nested_concat` flag.

**Notable Dependencies/Interfaces:**

* `nested_concat`: function for recursive tensor concatenation
* `nested_numpify`: function for converting tensors to numpy arrays

Example usage:
```python
container = EvalLoopContainer(do_nested_concat=True)
container.add(tensors1)
container.add(tensors2)
result = container.get_arrays()
```
Note: The exact implementation of `nested concat` and `nested numpify` functions are not included in this summary, as they are assumed to be separate modules or functions.


key: chroma:document
string_value:
CODE:

class SequentialDistributedSampler(Sampler):
    """
    Distributed Sampler that subsamples indices sequentially, making it easier to collate all results at the end.

    Even though we only use this sampler for eval and predict (no training), which means that the model params won't
    have to be synced (i.e. will not hang for synchronization even if varied number of forward passes), we still add
    extra samples to the sampler to make it evenly divisible (like in `DistributedSampler`) to make it easy to `gather`
    or `reduce` resulting tensors at the end of the loop.
    """

    def __init__(self, dataset, num_replicas=None, rank=None, batch_size=None):
        warnings.warn(
            "SequentialDistributedSampler is deprecated and will be removed in v5 of Transformers.",
            FutureWarning,
        )
        if num_replicas is None:
            if not dist.is_available():
                raise RuntimeError("Requires distributed package to be available")
            num_replicas = dist.get_world_size()
        if rank is None:
            if not dist.is_available():
                raise RuntimeError("Requires distributed package to be available")
            rank = dist.get_rank()
        self.dataset = dataset
        self.num_replicas = num_replicas
        self.rank = rank
        num_samples = len(self.dataset)
        # Add extra samples to make num_samples a multiple of batch_size if passed
        if batch_size is not None:
            self.num_samples = int(math.ceil(num_samples / (batch_size * num_replicas))) * batch_size
        else:
            self.num_samples = int(math.ceil(num_samples / num_replicas))
        self.total_size = self.num_samples * self.num_replicas
        self.batch_size = batch_size

    def __iter__(self):
        indices = list(range(len(self.dataset)))

        # add extra samples to make it evenly divisible
        indices += indices[: (self.total_size - len(indices))]
        assert (
            len(indices) == self.total_size
        ), f"Indices length {len(indices)} and total size {self.total_size} mismatched"

        # subsample
        indices = indices[self.rank * self.num_samples : (self.rank + 1) * self.num_samples]
        assert (
            len(indices) == self.num_samples
        ), f"Indices length {len(indices)} and sample number {self.num_samples} mismatched"

        return iter(indices)

    def __len__(self):


SUMMARY:
**SequentialDistributedSampler**

A distributed sampler for subsampling indices sequentially, ensuring evenly divisible samples for easy collation.

### Primary Purpose

* Subsample indices from a dataset to facilitate even distribution and easier collation of results during evaluation.

### Key Implementation Details

* Distributes indices among `num_replicas` using `rank`-based ordering.
* Ensures `num_samples` is a multiple of batch size (`batch_size`) when specified.

### Notable Dependencies or Requirements

* Distributed package (e.g., PyTorch, DDP)
* Available dimensions:
	+ `dataset`: input dataset object
	+ `dist`: distributed package interface (PyTorch, torch.distributed)

### Initializer Details

* Takes in:
	+ `dataset`: input dataset object
	+ `num_replicas` (optional): number of replicas to distribute indices among
	+ `rank` (optional): current rank for indexing distribution
	+ `batch_size` (optional): batch size for subsampling, affecting total sample count

### Sampler Operations

* Returns an iterator producing indices with a rank-ordered sequence.
* Employs list concatenation and slicing to add extra samples for even distribution.


key: chroma:document
string_value:
CODE:

def get_tpu_sampler(dataset: torch.utils.data.Dataset, batch_size: int):
    if xm.xrt_world_size() <= 1:
        return RandomSampler(dataset)


SUMMARY:
**get_tpu_sampler Function Summary**

**Purpose and Functionality**
This function returns a sampler for generating batches from a given `torch.utils.data.Dataset`. The sampler is optimized for multi-gpu (TPU) distributed training.

**Key Implementation Details:**

* Returns either a `RandomSampler` or a `DistributedSampler` depending on the number of workers (`xrt_world_size()`).
	+ If one worker, returns `RandomSampler`.
	+ Otherwise, returns `DistributedSampler` with customized number of replicas and rank.

**Notable Dependencies and Requirements:**

* `torch.utils.data.Dataset`
* `torch.distributed` (specifically, `xm` module)
* External dependency:
	+ PyTorch library (not explicitly listed in the code snippet)

**Notes:** The function assumes access to TPU (Tensor Processing Unit) resources for distributed training.


key: chroma:document
string_value:
CODE:

def nested_new_like(arrays, num_samples, padding_index=-100):
    """Create the same nested structure as `arrays` with a first dimension always at `num_samples`."""
    if isinstance(arrays, (list, tuple)):
        return type(arrays)(nested_new_like(x, num_samples) for x in arrays)


SUMMARY:
**Summary**

The `nested_new_like` function creates a new array with the same nested structure as `arrays`, but with the first dimension always of length `num_samples`. It handles recursive nesting and inserts padding (`-100`) if necessary.

**Key Details:**

*   Creates a nested array by applying the function to each element in `arrays`.
*   Fills any non-full arrays (1st dimension < num\_samples) with padding `-100`.

**Implementation:**

*   Returns:
    *   A new array with the same shape as `arrays`, but with the first dimension expanded to length `num_samples`.
    *   If `arrays` is not a nested structure, it fills only the requested dimensions (`shape[1:]`) with padding.


key: chroma:document
string_value:
CODE:

def expand_like(arrays, new_seq_length, padding_index=-100):
    """Expand the `arrays` so that the second dimension grows to `new_seq_length`. Uses `padding_index` for padding."""
    result = np.full_like(arrays, padding_index, shape=(arrays.shape[0], new_seq_length) + arrays.shape[2:])
    result[:, : arrays.shape[1]] = arrays


SUMMARY:
**Summary:**

`expand_like` function expands the second dimension of input `arrays` to `new_seq_length`. 
It pads with `-100` and fills first columns with original data.

**Key Implementation Details:**

- Uses `padding_index` for padding
- Creates a new array with shape `(shape[0], new_seq_length, ...)` filled with `padding_index`
- Fills first `arrays.shape[1]` columns with original data from `arrays`

**Dependencies and Requirements:**

- Import of `numpy` library for `np.full_like` function
- Input `arrays` is expected to have shape `(shape[0], ...)` where `...` represents additional dimensions


key: chroma:document
string_value:
CODE:

def nested_truncate(tensors, limit):
    "Truncate `tensors` at `limit` (even if it's a nested list/tuple/dict of tensors)."
    if isinstance(tensors, (list, tuple)):
        return type(tensors)(nested_truncate(t, limit) for t in tensors)
    if isinstance(tensors, Mapping):
        return type(tensors)({k: nested_truncate(t, limit) for k, t in tensors.items()})



SUMMARY:
**Code Summary**

`nested_truncate` function truncates input `tensors` (list, tuple, dict, or list/dict elements) to a specified `limit`. Recursively applies truncation to nested data structures.

**Implementation Details**

*   Truncates elements of type list, tuple, and Mapping
*   Uses slicing (`tensors[:limit]`) for flat lists/tuples

**Key Methods/Ideals**

*   Used with tensors, which can be recursively defined (e.g., `type(tensors)`)
*   Inherited/implemented interfaces: None


key: chroma:document
string_value:
CODE:

class DistributedTensorGatherer:
    """
    A class responsible for properly gathering tensors (or nested list/tuple of tensors) on the CPU by chunks.

    If our dataset has 16 samples with a batch size of 2 on 3 processes and we gather then transfer on CPU at every
    step, our sampler will generate the following indices:

        `[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1]`

    to get something of size a multiple of 3 (so that each process gets the same dataset length). Then process 0, 1 and
    2 will be responsible of making predictions for the following samples:

        - P0: `[0, 1, 2, 3, 4, 5]`
        - P1: `[6, 7, 8, 9, 10, 11]`
        - P2: `[12, 13, 14, 15, 0, 1]`

    The first batch treated on each process will be

        - P0: `[0, 1]`
        - P1: `[6, 7]`
        - P2: `[12, 13]`

    So if we gather at the end of the first batch, we will get a tensor (nested list/tuple of tensor) corresponding to
    the following indices:

        `[0, 1, 6, 7, 12, 13]`

    If we directly concatenate our results without taking any precautions, the user will then get the predictions for
    the indices in this order at the end of the prediction loop:

        `[0, 1, 6, 7, 12, 13, 2, 3, 8, 9, 14, 15, 4, 5, 10, 11, 0, 1]`

    For some reason, that's not going to roll their boat. This class is there to solve that problem.

    Args:
        world_size (`int`):
            The number of processes used in the distributed training.
        num_samples (`int`):
            The number of samples in our dataset.
        make_multiple_of (`int`, *optional*):
            If passed, the class assumes the datasets passed to each process are made to be a multiple of this argument
            (by adding samples).
        padding_index (`int`, *optional*, defaults to -100):
            The padding index to use if the arrays don't all have the same sequence length.
    """

    def __init__(self, world_size, num_samples, make_multiple_of=None, padding_index=-100):
        warnings.warn(
            "DistributedTensorGatherer is deprecated and will be removed in v5 of Transformers.",
            FutureWarning,
        )
        self.world_size = world_size
        self.num_samples = num_samples
        total_size = world_size if make_multiple_of is None else world_size * make_multiple_of
        self.total_samples = int(np.ceil(num_samples / total_size)) * total_size
        self.process_length = self.total_samples // world_size
        self._storage = None
        self._offsets = None
        self.padding_index = padding_index

    def add_arrays(self, arrays):
        """
        Add `arrays` to the internal storage, Will initialize the storage to the full size at the first arrays passed
        so that if we're bound to get an OOM, it happens at the beginning.
        """
        if arrays is None:
            return
        if self._storage is None:
            self._storage = nested_new_like(arrays, self.total_samples, padding_index=self.padding_index)
            self._offsets = list(range(0, self.total_samples, self.process_length))

        slice_len, self._storage = self._nested_set_tensors(self._storage, arrays)
        for i in range(self.world_size):
            self._offsets[i] += slice_len

    def _nested_set_tensors(self, storage, arrays):
        if isinstance(arrays, (list, tuple)):
            result = [self._nested_set_tensors(x, y) for x, y in zip(storage, arrays)]
            return result[0][0], type(arrays)(r[1] for r in result)
        assert (
            arrays.shape[0] % self.world_size == 0
        ), f"Arrays passed should all have a first dimension multiple of {self.world_size}, found {arrays.shape[0]}."

        slice_len = arrays.shape[0] // self.world_size
        for i in range(self.world_size):
            if len(arrays.shape) == 1:
                storage[self._offsets[i] : self._offsets[i] + slice_len] = arrays[i * slice_len : (i + 1) * slice_len]
            else:
                # Expand the array on the fly if needed.
                if len(storage.shape) > 1 and storage.shape[1] < arrays.shape[1]:
                    storage = expand_like(storage, arrays.shape[1], padding_index=self.padding_index)
                storage[self._offsets[i] : self._offsets[i] + slice_len, : arrays.shape[1]] = arrays[
                    i * slice_len : (i + 1) * slice_len
                ]
        return slice_len, storage

    def finalize(self):
        """
        Return the properly gathered arrays and truncate to the number of samples (since the sampler added some extras
        to get each process a dataset of the same length).
        """
        if self._storage is None:
            return
        if self._offsets[0] != self.process_length:
            logger.warning("Not all data has been set. Are you sure you passed all values?")


SUMMARY:
**DistributedTensorGatherer Class Summary**

### Main Purpose and Functionality

The `DistributedTensorGatherer` class is designed to efficiently gather tensors on the CPU by chunks in a distributed environment. It ensures that each process receives data of the same length, avoiding interleaving predictions from different samples.

### Key Implementation Details

* The class initializes internal storage for gathering arrays using `nested_new_like`.
* The `_nested_set_tensors` method distributes arrays across processes, handling single-dimension and multi-dimensional arrays.
* The `finalize` method truncates the gathered data to the correct number of samples after adding padding indices.

### Notable Dependencies or Requirements

* The class uses NumPy (`np`) for tensor operations and warnings logging (`logger`).
* No external dependencies are required.
* The class implements interfaces based on custom `nested_new_like`, `nested_truncate`, and `expand_like` functions, which are assumed to be defined elsewhere in the codebase.

### Implementation Signature

```markdown
class DistributedTensorGatherer:
    def __init__(self, world_size, num_samples, make_multiple_of=None, padding_index=-100):
        pass
    
    def add_arrays(self, arrays):
        pass
    
    def _nested_set_tensors(self, storage, arrays):
        pass
    
    def finalize(self):
        pass
```


key: chroma:document
string_value:
CODE:

def get_length_grouped_indices(lengths, batch_size, mega_batch_mult=None, generator=None):
    """
    Return a list of indices so that each slice of `batch_size` consecutive indices correspond to elements of similar
    lengths. To do this, the indices are:

    - randomly permuted
    - grouped in mega-batches of size `mega_batch_mult * batch_size`
    - sorted by length in each mega-batch

    The result is the concatenation of all mega-batches, with the batch of `batch_size` containing the element of
    maximum length placed first, so that an OOM happens sooner rather than later.
    """
    # Default for mega_batch_mult: 50 or the number to get 4 megabatches, whichever is smaller.
    if mega_batch_mult is None:
        mega_batch_mult = min(len(lengths) // (batch_size * 4), 50)
        # Just in case, for tiny datasets
        if mega_batch_mult == 0:
            mega_batch_mult = 1

    # We need to use torch for the random part as a distributed sampler will set the random seed for torch.
    indices = torch.randperm(len(lengths), generator=generator)
    megabatch_size = mega_batch_mult * batch_size
    megabatches = [indices[i : i + megabatch_size].tolist() for i in range(0, len(lengths), megabatch_size)]
    megabatches = [sorted(megabatch, key=lambda i: lengths[i], reverse=True) for megabatch in megabatches]

    # The rest is to get the biggest batch first.
    # Since each megabatch is sorted by descending length, the longest element is the first
    megabatch_maximums = [lengths[megabatch[0]] for megabatch in megabatches]
    max_idx = torch.argmax(torch.tensor(megabatch_maximums)).item()
    # Switch to put the longest element in first position
    megabatches[0][0], megabatches[max_idx][0] = megabatches[max_idx][0], megabatches[0][0]



SUMMARY:
Here is a concise summary of the provided function:

**Function:** `get_length_grouped_indices`

**Purpose:** Generate a list of indices by randomly permuting and grouping `batch_size` consecutive indices into mega-batches based on element lengths.

**Key Implementation:**

* Randomly permutes indices using `torch.randperm`
* Groups indices into mega-batches based on `mega_batch_mult * batch_size`
* Sorts each mega-batch by descending length
* Prioritizes the longest element in the first position of each mega-batch

**Notable Dependencies and Requirements:**

* pytorch (`torch`); requires a GPU-based distributed sampler for random seed generation.
* `generator` (optional): allows external seed management.


key: chroma:document
string_value:
CODE:

class LengthGroupedSampler(Sampler):
    r"""
    Sampler that samples indices in a way that groups together features of the dataset of roughly the same length while
    keeping a bit of randomness.
    """

    def __init__(
        self,
        batch_size: int,
        dataset: Optional[Dataset] = None,
        lengths: Optional[List[int]] = None,
        model_input_name: Optional[str] = None,
        generator=None,
    ):
        if dataset is None and lengths is None:
            raise ValueError("One of dataset and lengths must be provided.")

        self.batch_size = batch_size
        if lengths is None:
            model_input_name = model_input_name if model_input_name is not None else "input_ids"
            if (
                not (isinstance(dataset[0], dict) or isinstance(dataset[0], BatchEncoding))
                or model_input_name not in dataset[0]
            ):
                raise ValueError(
                    "Can only automatically infer lengths for datasets whose items are dictionaries with an "
                    f"'{model_input_name}' key."
                )
            lengths = [len(feature[model_input_name]) for feature in dataset]
        elif isinstance(lengths, torch.Tensor):
            logger.info(
                "If lengths is a torch.Tensor, LengthGroupedSampler will be slow. Converting lengths to List[int]..."
            )
            lengths = lengths.tolist()

        self.lengths = lengths
        self.generator = generator

    def __len__(self):
        return len(self.lengths)

    def __iter__(self):
        indices = get_length_grouped_indices(self.lengths, self.batch_size, generator=self.generator)


SUMMARY:
**LengthGroupedSampler Class Summary**

**Main Purpose and Functionality:**
The `LengthGroupedSampler` class is a sampler that groups indices from a dataset based on feature lengths while maintaining randomization.

**Key Implementation Details:**

* Employs the `get_length_grouped_indices` function to generate indices.
* Infers feature lengths automatically for datasets with dictionary-like structures and specified input name.
* Raises errors if either `dataset` or `lengths` are missing.
* Includes an optional generator parameter.

**Notable Dependencies and Requirements:**

* `torch` library (version 1.9+)
* `get_length_grouped_indices` function (not provided in the chunk)

**Method Signatures and Interfaces:**

* `__init__(batch_size, dataset, lengths, model_input_name, generator)`
* `__len__()`
* `__iter__()`

Note that the `get_length_grouped_indices` function is not included in this summary as its implementation details are not provided.


key: chroma:document
string_value:
CODE:

class DistributedLengthGroupedSampler(DistributedSampler):
    r"""
    Distributed Sampler that samples indices in a way that groups together features of the dataset of roughly the same
    length while keeping a bit of randomness.
    """

    # Copied and adapted from PyTorch DistributedSampler.
    def __init__(
        self,
        batch_size: int,
        dataset: Optional[Dataset] = None,
        num_replicas: Optional[int] = None,
        rank: Optional[int] = None,
        seed: int = 0,
        drop_last: bool = False,
        lengths: Optional[List[int]] = None,
        model_input_name: Optional[str] = None,
    ):
        if dataset is None and lengths is None:
            raise ValueError("One of dataset and lengths must be provided.")
        if num_replicas is None:
            if not dist.is_available():
                raise RuntimeError("Requires distributed package to be available")
            num_replicas = dist.get_world_size()
        if rank is None:
            if not dist.is_available():
                raise RuntimeError("Requires distributed package to be available")
            rank = dist.get_rank()

        self.batch_size = batch_size
        self.num_replicas = num_replicas
        self.rank = rank
        self.epoch = 0
        self.drop_last = drop_last

        if lengths is None:
            model_input_name = model_input_name if model_input_name is not None else "input_ids"
            if (
                not (isinstance(dataset[0], dict) or isinstance(dataset[0], BatchEncoding))
                or model_input_name not in dataset[0]
            ):
                raise ValueError(
                    "Can only automatically infer lengths for datasets whose items are dictionaries with an "
                    f"'{model_input_name}' key."
                )
            lengths = [len(feature[model_input_name]) for feature in dataset]
        elif isinstance(lengths, torch.Tensor):
            logger.info(
                "If lengths is a torch.Tensor, DistributedLengthGroupedSampler will be slow. Converting lengths to"
                " List[int]..."
            )
            lengths = lengths.tolist()

        self.lengths = lengths

        # If the dataset length is evenly divisible by # of replicas, then there
        # is no need to drop any data, since the dataset will be split equally.
        if self.drop_last and len(self.lengths) % self.num_replicas != 0:
            # Split to nearest available length that is evenly divisible.
            # This is to ensure each rank receives the same amount of data when
            # using this Sampler.
            self.num_samples = math.ceil((len(self.lengths) - self.num_replicas) / self.num_replicas)
        else:
            self.num_samples = math.ceil(len(self.lengths) / self.num_replicas)
        self.total_size = self.num_samples * self.num_replicas
        self.seed = seed

    def __iter__(self) -> Iterator:
        # Deterministically shuffle based on epoch and seed
        g = torch.Generator()
        g.manual_seed(self.seed + self.epoch)
        indices = get_length_grouped_indices(self.lengths, self.batch_size, generator=g)

        if not self.drop_last:
            # add extra samples to make it evenly divisible
            indices += indices[: (self.total_size - len(indices))]
        else:
            # remove tail of data to make it evenly divisible.
            indices = indices[: self.total_size]
        assert len(indices) == self.total_size

        # subsample
        indices = indices[self.rank : self.total_size : self.num_replicas]
        assert len(indices) == self.num_samples



SUMMARY:
**Distributed Length Grouped Sampler Class Summary**

**Main Purpose and Functionality:**
This class is a customized `DistributedSampler` that groups together features of the dataset with roughly similar lengths while maintaining some randomness.

**Key Implementation Details:**

* Sampling indices using `get_length_grouped_indices` with epoch, seed, batch size, and data lengths.
* Deterministically shuffles based on epoch and seed.
* Groups samples into batches using a random permutation.
* Adjusts batch sizes to make the total number of samples evenly divisible by the number of replicas.

**Notable Dependencies or Requirements:**

* `torch` (for tensor operations)
* `dist` (for distributed package and replica indices)

**Interfaces and Inherited Dependencies:**

* `Dataset` interface (for automatically inferring data lengths)
* `BatchEncoding` class (for checking dataset item format)


key: chroma:document
string_value:
CODE:

class ShardSampler(Sampler):
    """
    Sampler that shards batches between several processes. Dispatches indices batch by batch: on 2 processes with batch
    size 4, the first two batches are `[0, 1, 2, 3, 4, 5, 6, 7]` and `[8, 9, 10, 11, 12, 13, 14, 15]`, which shard into
    `[0, 1, 2, 3]` and `[8, 9, 10, 11]` for GPU-0 and `[4, 5, 6, 7]` and `[12, 13, 14, 15]` for GPU-1.

    The sampler thus yields `[0, 1, 2, 3, 8, 9, 10, 11]` on GPU-0 and `[4, 5, 6, 7, 12, 13, 14, 15]` on GPU-1.
    """

    def __init__(
        self,
        dataset: Dataset,
        batch_size: int = 1,
        drop_last: bool = False,
        num_processes: int = 1,
        process_index: int = 0,
    ):
        self.dataset = dataset
        self.batch_size = batch_size
        self.drop_last = drop_last
        self.num_processes = num_processes
        self.process_index = process_index

        self.total_batch_size = total_batch_size = batch_size * num_processes

        num_batches = len(dataset) // total_batch_size if drop_last else math.ceil(len(dataset) / total_batch_size)
        self.total_num_samples = num_batches * total_batch_size

    def __iter__(self):
        indices = list(range(len(self.dataset)))

        # Add extra samples to make it evenly divisible. While loop is there in the edge case we have a tiny dataset
        # and it needs to be done several times.
        while len(indices) < self.total_num_samples:
            indices += indices[: (self.total_num_samples - len(indices))]

        result = []
        for batch_start in range(self.batch_size * self.process_index, self.total_num_samples, self.total_batch_size):
            result += indices[batch_start : batch_start + self.batch_size]

        return iter(result)

    def __len__(self):
        # Each shard only sees a fraction of total_num_samples.


SUMMARY:
**ShardSampler Class Summary**

*   **Purpose**: Shard batches between several processes, dispatching indices in parallel to multiple GPUs.

### Key Implementation Details

*   Shards dataset into `num_processes` number of queues and assigns them to `num_processes` individual processes (GPU-0 to GPU-`process_index`)
*   Dispatches batch by batch; first `batch_size * process_index` batches go to the current process.
*   Calculates the total number of samples per queue based on the dataset size and batch size.
*   Iterates over each batch, assigning indices from the main list, repeated as needed to fill the queue.

### Notable Dependencies/Requirements

*   `Sampler`: Base class for sharding algorithm
*   `Dataset`: Provides data for sharding; inherits from this class to access dataset properties (e.g., `__len__`)
*   External dependencies:
    *   `math` (for ceil function)
    *   Python's built-in `range` and list concatenation functions (`+`).


key: chroma:document
string_value:
CODE:

class IterableDatasetShard(IterableDataset):
    """
    Wraps a PyTorch `IterableDataset` to generate samples for one of the processes only. Instances of this class will
    always yield a number of samples that is a round multiple of the actual batch size (which is `batch_size x
    num_processes`). Depending on the value of the `drop_last` attribute, it will either stop the iteration at the
    first batch that would be too small or loop with indices from the beginning.

    On two processes with an iterable dataset yielding of `[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]` with a batch size of
    2:

    - the shard on process 0 will yield `[0, 1, 4, 5, 8, 9]` so will see batches `[0, 1]`, `[4, 5]`, `[8, 9]`
    - the shard on process 1 will yield `[2, 3, 6, 7, 10, 11]` so will see batches `[2, 3]`, `[6, 7]`, `[10, 11]`

    <Tip warning={true}>

        If your IterableDataset implements some randomization that needs to be applied the same way on all processes
        (for instance, a shuffling), you should use a `torch.Generator` in a `generator` attribute of the `dataset` to
        generate your random numbers and call the [`~trainer_pt_utils.IterableDatasetShard.set_epoch`] method of this
        object. It will set the seed of this `generator` to `seed + epoch` on all processes before starting the
        iteration. Alternatively, you can also implement a `set_epoch()` method in your iterable dataset to deal with
        this.

    </Tip>

    Args:
        dataset (`torch.utils.data.IterableDataset`):
            The batch sampler to split in several shards.
        batch_size (`int`, *optional*, defaults to 1):
            The size of the batches per shard.
        drop_last (`bool`, *optional*, defaults to `False`):
            Whether or not to drop the last incomplete batch or complete the last batches by using the samples from the
            beginning.
        num_processes (`int`, *optional*, defaults to 1):
            The number of processes running concurrently.
        process_index (`int`, *optional*, defaults to 0):
            The index of the current process.
        seed (`int`, *optional*, defaults to 0):
            A random seed that will be used for the random number generation in
            [`~trainer_pt_utils.IterableDatasetShard.set_epoch`].
    """

    def __init__(
        self,
        dataset: IterableDataset,
        batch_size: int = 1,
        drop_last: bool = False,
        num_processes: int = 1,
        process_index: int = 0,
        seed: int = 0,
    ):
        self.dataset = dataset
        self.batch_size = batch_size
        self.drop_last = drop_last
        self.num_processes = num_processes
        self.process_index = process_index
        self.seed = seed
        self.epoch = 0
        self.num_examples = 0

    def set_epoch(self, epoch):
        self.epoch = epoch
        if hasattr(self.dataset, "set_epoch"):
            self.dataset.set_epoch(epoch)

    def __iter__(self):
        self.num_examples = 0
        if (
            not hasattr(self.dataset, "set_epoch")
            and hasattr(self.dataset, "generator")
            and isinstance(self.dataset.generator, torch.Generator)
        ):
            self.dataset.generator.manual_seed(self.seed + self.epoch)
        real_batch_size = self.batch_size * self.num_processes
        process_slice = range(self.process_index * self.batch_size, (self.process_index + 1) * self.batch_size)

        first_batch = None
        current_batch = []
        for element in self.dataset:
            self.num_examples += 1
            current_batch.append(element)
            # Wait to have a full batch before yielding elements.
            if len(current_batch) == real_batch_size:
                for i in process_slice:
                    yield current_batch[i]
                if first_batch is None:
                    first_batch = current_batch.copy()
                current_batch = []

        # Finished if drop_last is True, otherwise complete the last batch with elements from the beginning.
        if not self.drop_last and len(current_batch) > 0:
            if first_batch is None:
                first_batch = current_batch.copy()
            while len(current_batch) < real_batch_size:
                current_batch += first_batch
            for i in process_slice:
                yield current_batch[i]

    def __len__(self):
        # Will raise an error if the underlying dataset is not sized.
        if self.drop_last:
            return (len(self.dataset) // (self.batch_size * self.num_processes)) * self.batch_size
        else:


SUMMARY:
**Summary**

`IterableDatasetShard` class: Wraps an `IterableDataset` to yield samples for a specific process. 

*   Purpose:
    *   Split an iterable dataset into shards based on batch size and processes.
    *   Yield batches in a round multiple of the actual batch size, depending on the `drop_last` attribute.
*   Key Implementation Details:
    *   Uses PyTorch `torch.Generator` for randomization if available.
    *   Sets seed for generator based on process index and epoch number.
    *   Yields samples from the dataset in chunks of `batch_size x num_processes`.
*   Notable Dependencies:
    *   `torch.utils.data.IterableDataset`
    *   `torch.Generator`
*   Interfaces:
    *   Implements PyTorch interface for `IterableDataset`

**Methods**

*   `__iter__()`: Iterates over the dataset, yielding samples in chunks of `batch_size x num_processes`.
*   `set_epoch(epoch)`: Sets the epoch number and updates the seed for randomization if applicable.
*   `__len__()`: Returns the size of the dataset based on the batch size and processes.

**Attributes**

*   `dataset`: The underlying iterable dataset.
*   `batch_size`: The size of batches per shard.
*   `drop_last`: Whether to drop the last incomplete batch or complete it with samples from the beginning.
*   `num_processes`: The number of processes running concurrently.
*   `process_index`: The index of the current process.
*   `seed`: A random seed used for randomization.
*   `epoch`: The epoch number.
*   `num_examples`: The number of examples yielded so far.


key: docstring
string_value:
# In order to keep `trainer.py` compact and easy to understand, place any secondary PT Trainer
# helper methods here


key: chroma:document
string_value:
CODE:

def _get_learning_rate(self):
    if self.is_deepspeed_enabled:
        # with deepspeed's fp16 and dynamic loss scale enabled the optimizer/scheduler steps may
        # not run for the first few dozen steps while loss scale is too large, and thus during
        # that time `get_last_lr` will fail if called during that warm up stage, so work around it:
        try:
            last_lr = self.lr_scheduler.get_last_lr()[0]
        except AssertionError as e:
            if "need to call step" in str(e):
                logger.warning("tried to get lr value before scheduler/optimizer started stepping, returning lr=0")
                last_lr = 0
            else:
                raise
    else:
        if isinstance(self.lr_scheduler, torch.optim.lr_scheduler.ReduceLROnPlateau):
            last_lr = self.optimizer.param_groups[0]["lr"]
        else:
            last_lr = self.lr_scheduler.get_last_lr()[0]
        if torch.is_tensor(last_lr):
            last_lr = last_lr.item()


SUMMARY:
**Function Summary: `_get_learning_rate`**

### Purpose and Functionality

This function retrieves the last learning rate value from an instance of the class. If DeepSpeed is enabled, it calculates the adjusted learning rate to avoid potential errors during warm-up stages. Otherwise, it extracts the latest learning rate from either the optimizer or the lr_scheduler.

**Key Implementation Details:**

* Uses `try`-`except` block to handle potential issues with `get_last_lr` when DeepSpeed is enabled.
* Checks if `lr_scheduler` is a `ReduceLROnPlateau` instance and extracts the current learning rate accordingly.
* Converts `last_lr` to a Python scalar value if it's a tensor.

**Notable Dependencies:**

* `torch.optim.lr_scheduler.ReduceLROnPlateau`
* External dependencies not specified in this function snippet.


key: chroma:document
string_value:
CODE:

def _secs2timedelta(secs):
    """
    convert seconds to hh:mm:ss.msec, msecs rounded to 2 decimals
    """

    msec = int(abs(secs - int(secs)) * 100)


SUMMARY:
**Function Summary**

### `_secs2timedelta`

Converts seconds to `hh:mm:ss.msec` format.

#### Key Implementation Details

* Uses `datetime.timedelta` to convert seconds to timedelta object
* Rounds milliseconds to 2 decimal places using integer division and string formatting

#### Notable Dependencies/Requirements

* `datetime` module (specifically, `timedelta` class)


key: chroma:document
string_value:
CODE:

def metrics_format(self, metrics: Dict[str, float]) -> Dict[str, float]:
    """
    Reformat Trainer metrics values to a human-readable format

    Args:
        metrics (`Dict[str, float]`):
            The metrics returned from train/evaluate/predict

    Returns:
        metrics (`Dict[str, float]`): The reformatted metrics
    """

    metrics_copy = metrics.copy()
    for k, v in metrics_copy.items():
        if "_mem_" in k:
            metrics_copy[k] = f"{ v >> 20 }MB"
        elif "_runtime" in k:
            metrics_copy[k] = _secs2timedelta(v)
        elif k == "total_flos":
            metrics_copy[k] = f"{ int(v) >> 30 }GF"
        elif isinstance(metrics_copy[k], float):
            metrics_copy[k] = round(v, 4)



SUMMARY:
**Summary:**

Reformats Trainer metrics values to a human-readable format.

```python
def metrics_format(self, metrics: Dict[str, float]) -> Dict[str, float]:
    """
    Reformat Trainer metrics values to a human-readable format
    
    Args:
        metrics (Dict[str, float]):
            Metrics returned from train/evaluate/predict
    
    Returns:
        metrics (Dict[str, float]): Reformatted metrics
    """

    reformatted_metrics = metrics.copy()
    
    # Convert memory usage and total FLOPs to human-readable format
    for key in ["_mem_", "_runtime", "total_flops"]:
        if key in metrics_copy:
            value = metrics_copy.pop(key)
            if key == "_mem_":
                reformatted_metrics[f"{key}{value}"] = f"{ value >> 20 }MB"
            elif key == "_runtime":
                reformatted_metrics[k] = _secs2timedelta(value)
            elif key == "total_flops":
                reformatted_metrics[k] = f"{ int(value) >> 30 }GF"
    
    # Round all other values to 4 decimal places
    for key, value in metrics_copy.items():
        if not isinstance(value, (int, float)):
            continue
        reformatted_metrics[key] = round(value, 4)
        
    return metrics_copy
```

**Key Implementation Details:**

- Uses bitwise right shift (`>>`) to reformat memory usage and FLOPs values.
- Applies `_secs2timedelta` function to format runtime value.
- Rounds all other float values to 4 decimal places.

**Notable Dependencies or Requirements:**

- `_secs2timedelta` function (not shown in this snippet).
- `float`, `int`, and `dict` types.


key: chroma:document
string_value:
CODE:

def log_metrics(self, split, metrics):
    """
    Log metrics in a specially formatted way

    Under distributed environment this is done only for a process with rank 0.

    Args:
        split (`str`):
            Mode/split name: one of `train`, `eval`, `test`
        metrics (`Dict[str, float]`):
            The metrics returned from train/evaluate/predictmetrics: metrics dict

    Notes on memory reports:

    In order to get memory usage report you need to install `psutil`. You can do that with `pip install psutil`.

    Now when this method is run, you will see a report that will include: :

    ```
    init_mem_cpu_alloc_delta   =     1301MB
    init_mem_cpu_peaked_delta  =      154MB
    init_mem_gpu_alloc_delta   =      230MB
    init_mem_gpu_peaked_delta  =        0MB
    train_mem_cpu_alloc_delta  =     1345MB
    train_mem_cpu_peaked_delta =        0MB
    train_mem_gpu_alloc_delta  =      693MB
    train_mem_gpu_peaked_delta =        7MB
    ```

    **Understanding the reports:**

    - the first segment, e.g., `train__`, tells you which stage the metrics are for. Reports starting with `init_`
        will be added to the first stage that gets run. So that if only evaluation is run, the memory usage for the
        `__init__` will be reported along with the `eval_` metrics.
    - the third segment, is either `cpu` or `gpu`, tells you whether it's the general RAM or the gpu0 memory
        metric.
    - `*_alloc_delta` - is the difference in the used/allocated memory counter between the end and the start of the
        stage - it can be negative if a function released more memory than it allocated.
    - `*_peaked_delta` - is any extra memory that was consumed and then freed - relative to the current allocated
        memory counter - it is never negative. When you look at the metrics of any stage you add up `alloc_delta` +
        `peaked_delta` and you know how much memory was needed to complete that stage.

    The reporting happens only for process of rank 0 and gpu 0 (if there is a gpu). Typically this is enough since the
    main process does the bulk of work, but it could be not quite so if model parallel is used and then other GPUs may
    use a different amount of gpu memory. This is also not the same under DataParallel where gpu0 may require much more
    memory than the rest since it stores the gradient and optimizer states for all participating GPUS. Perhaps in the
    future these reports will evolve to measure those too.

    The CPU RAM metric measures RSS (Resident Set Size) includes both the memory which is unique to the process and the
    memory shared with other processes. It is important to note that it does not include swapped out memory, so the
    reports could be imprecise.

    The CPU peak memory is measured using a sampling thread. Due to python's GIL it may miss some of the peak memory if
    that thread didn't get a chance to run when the highest memory was used. Therefore this report can be less than
    reality. Using `tracemalloc` would have reported the exact peak memory, but it doesn't report memory allocations
    outside of python. So if some C++ CUDA extension allocated its own memory it won't be reported. And therefore it
    was dropped in favor of the memory sampling approach, which reads the current process memory usage.

    The GPU allocated and peak memory reporting is done with `torch.cuda.memory_allocated()` and
    `torch.cuda.max_memory_allocated()`. This metric reports only "deltas" for pytorch-specific allocations, as
    `torch.cuda` memory management system doesn't track any memory allocated outside of pytorch. For example, the very
    first cuda call typically loads CUDA kernels, which may take from 0.5 to 2GB of GPU memory.

    Note that this tracker doesn't account for memory allocations outside of [`Trainer`]'s `__init__`, `train`,
    `evaluate` and `predict` calls.

    Because `evaluation` calls may happen during `train`, we can't handle nested invocations because
    `torch.cuda.max_memory_allocated` is a single counter, so if it gets reset by a nested eval call, `train`'s tracker
    will report incorrect info. If this [pytorch issue](https://github.com/pytorch/pytorch/issues/16266) gets resolved
    it will be possible to change this class to be re-entrant. Until then we will only track the outer level of
    `train`, `evaluate` and `predict` methods. Which means that if `eval` is called during `train`, it's the latter
    that will account for its memory usage and that of the former.

    This also means that if any other tool that is used along the [`Trainer`] calls
    `torch.cuda.reset_peak_memory_stats`, the gpu peak memory stats could be invalid. And the [`Trainer`] will disrupt
    the normal behavior of any such tools that rely on calling `torch.cuda.reset_peak_memory_stats` themselves.

    For best performance you may want to consider turning the memory profiling off for production runs.
    """
    if not self.is_world_process_zero():
        return

    print(f"***** {split} metrics *****")
    metrics_formatted = self.metrics_format(metrics)
    k_width = max(len(str(x)) for x in metrics_formatted.keys())
    v_width = max(len(str(x)) for x in metrics_formatted.values())
    for key in sorted(metrics_formatted.keys()):


SUMMARY:
**Function Summary: `log_metrics`**

**Purpose and Functionality**
The `log_metrics` function logs metrics in a specially formatted way, providing detailed memory usage reports for CPU and GPU.

**Key Implementation Details**

*   Logs memory reports for each stage (init, train, eval, test) under a distributed environment, where the process rank 0 is considered.
*   Uses `psutil` to calculate memory usage.
*   Calculates memory allocations and releases using Python's GIL (`psutil` or `tracemalloc`) for CPU and PyTorch-specific CUDA calls for GPU.

**Notable Dependencies or Requirements**

*   `torch` library (depending on GPU support)
*   `psutil` library for memory usage tracking
*   `pytorch` issue [16266](https://github.com/pytorch/pytorch/issues/16266) regarding memory allocation

**Interface and Parameters**

*   `split` (`str`) - Mode/split name: one of `train`, `eval`, `test`
*   `metrics` (`Dict[str, float]`) - The metrics returned from train/evaluate/predict, which are formatted by the method.

**Design Choice and Context**
This function is used primarily for process rank 0 under a distributed environment. In case of model parallel or DataParallel use cases, GPU memory may need to be considered separately in these configurations.


key: chroma:document
string_value:
CODE:

def save_metrics(self, split, metrics, combined=True):
    """
    Save metrics into a json file for that split, e.g. `train_results.json`.

    Under distributed environment this is done only for a process with rank 0.

    Args:
        split (`str`):
            Mode/split name: one of `train`, `eval`, `test`, `all`
        metrics (`Dict[str, float]`):
            The metrics returned from train/evaluate/predict
        combined (`bool`, *optional*, defaults to `True`):
            Creates combined metrics by updating `all_results.json` with metrics of this call

    To understand the metrics please read the docstring of [`~Trainer.log_metrics`]. The only difference is that raw
    unformatted numbers are saved in the current method.

    """
    if not self.is_world_process_zero():
        return

    path = os.path.join(self.args.output_dir, f"{split}_results.json")
    with open(path, "w") as f:
        json.dump(metrics, f, indent=4, sort_keys=True)

    if combined:
        path = os.path.join(self.args.output_dir, "all_results.json")
        if os.path.exists(path):
            with open(path, "r") as f:
                all_metrics = json.load(f)
        else:
            all_metrics = {}

        all_metrics.update(metrics)
        with open(path, "w") as f:


SUMMARY:
**Function Summary: `save_metrics`**

*   **Purpose**: Saves metrics to a JSON file for a specified split (`train`, `eval`, `test`, or `all`) and optionally updates the combined metrics in `all_results.json`.

**Key Implementation Details:**

*   The function uses `json.dump` to serialize the `metrics` dictionary to the output directory.
*   In distributed environments, only processes with rank 0 commit to the JSON file.

**Notable Dependencies and Requirements:**

*   `os`: used for interacting with the operating system (output directory construction).
*   `json`: used for serializing/deserializing the metrics data.

**Method Signatures and Interfaces:**

*   The function does not inherit from any interfaces but uses the `Trainer` class's methods (`is_world_process_zero`, `args.output_dir`).


key: chroma:document
string_value:
CODE:

def save_state(self):
    """
    Saves the Trainer state, since Trainer.save_model saves only the tokenizer with the model

    Under distributed environment this is done only for a process with rank 0.
    """
    if not self.is_world_process_zero():
        return

    path = os.path.join(self.args.output_dir, "trainer_state.json")


SUMMARY:
**Summary**

The `save_state` function saves the trainer's state to a JSON file when running in a distributed environment. It is only executed on process rank 0, preserving the model's tokenizer when saved.

**Key Implementation Details**

* Saves trainer state to local file at `self.args.output_dir/trainer_state.json`
* Excludes writes for non-rank-0 processes

**Notable Dependencies/Requirements**

* `os.path.join` for constructing output path
* `self.state` (implies a `TrainerState` class or similar) for saving state to JSON
* `Trainer.save_model` (presumably on same model, not included due to context)
* Distributed environment setup utilizing process ranks (e.g., `multiprocessing` module is implied, though not explicitly mentioned)


key: chroma:document
string_value:
CODE:

def get_model_param_count(model, trainable_only=False):
    """
    Calculate model's total param count. If trainable_only is True then count only those requiring grads
    """
    if is_deepspeed_zero3_enabled():

        def numel(p):
            return p.ds_numel if hasattr(p, "ds_numel") else p.numel()

    else:

        def numel(p):
            return p.numel()



SUMMARY:
**Function Summary:**

`get_model_param_count(model, trainable_only=False)` calculates the total parameter count of the input `model`. If `trainable_only` is `True`, only parameters requiring gradients are counted.

**Key Implementation Details:**

- Uses DeepSpeed 3 (DS3) enabled models by default (`numel(p) = p.ds_numel if hasattr(p, "ds_numel") else p.numel()`).
- Checks for trainable parameters with `models.requires_grad`.

**Notable Dependencies/Requirements:**

- `model` parameter is assumed to be an instance of PyTorch model.
- DeepSpeed 3 (DS3) related classes/functions (`is_deepspeed_zero3_enabled()`, `ds_numel`) may be external dependencies.


key: chroma:document
string_value:
CODE:

def get_parameter_names(model, forbidden_layer_types):
    """
    Returns the names of the model parameters that are not inside a forbidden layer.
    """
    result = []
    for name, child in model.named_children():
        result += [
            f"{name}.{n}"
            for n in get_parameter_names(child, forbidden_layer_types)
            if not isinstance(child, tuple(forbidden_layer_types))
        ]
    # Add model specific parameters (defined with nn.Parameter) since they are not in any child.
    result += list(model._parameters.keys())


SUMMARY:
**Summary: `get_parameter_names` Function**

* **Purpose:** Returns the names of model parameters outside forbidden layer types.

**Key Implementation Details:**

* Recursively traverses the model's child nodes using `named_children()` method.
* Filters out parameters within specified forbidden layer types (`isinstance(child, tuple(forbidden_layer_types))`).
* Includes global model parameters not contained in any subtree by looking at `_parameters.keys()`.

**Notable Dependencies/Requirements:**

* Uses PyTorch's `nn.Parameter` and its related metadata (`_parameters`) internally.
* Does not utilize external dependencies.


key: chroma:document
string_value:
CODE:

def get_module_class_from_name(module, name):
    """
    Gets a class from a module by its name.

    Args:
        module (`torch.nn.Module`): The module to get the class from.
        name (`str`): The name of the class.
    """
    modules_children = list(module.children())
    if module.__class__.__name__ == name:
        return module.__class__
    elif len(modules_children) == 0:
        return
    else:
        for child_module in modules_children:
            module_class = get_module_class_from_name(child_module, name)
            if module_class is not None:


SUMMARY:
**Function Summary: `get_module_class_from_name`**

Gets a class from a PyTorch neural network module by its name.

* Primary purpose: Recursively search for a specific class within the given module and its children.
* Key implementation details:
	+ Iterates through the module's children using `module.children()`.
	+ Checks if the current module matches the target name, returning itself if so.
	+ Recursively calls itself on each child module until the target class is found or no more children exist.
* Notable dependencies and requirements: None
* Interface:
```
get_module_class_from_name(module: torch.nn.Module, name: str)
```

**Key Takeaways:**

* This function assumes that the input `module` is a valid PyTorch neural network module.
* The function returns the class of the matched module or `None` if no match is found.
* Recursive approach used to traverse the module's hierarchy.


key: chroma:document
string_value:
CODE:

def remove_dummy_checkpoint(is_main_process, output_dir, filenames):
    if is_main_process:
        for filename in filenames:
            file = os.path.join(output_dir, filename)
            if os.path.isfile(file):


SUMMARY:
**Concise Summary**

`remove_dummy_checkpoint` function removes dummy checkpoint files from an output directory when running as the main process.

**Detailed Analysis**

* **Purpose**: Removes unwanted files based on user-provided `filenames`.
* **Key Implementation Details**:
	+ Iterates over a list of file names using `os.path.join`.
	+ Checks if each generated file exists using `os.path.isfile` and removes it using `os.remove` when in main process.
* **Notable Dependencies or Requirements**: None.

**Summary Classifications**

|  | Main Purpose | Key Implementation Details |
| --- | --- | --- |
| 1 | Removes dummy checkpoint files | Iterates over file names, checks existence, and deletes them. |
| 2 | User-provided input required | Uses `os.path.join` to construct file paths. |


key: chroma:document
string_value:
CODE:

class LayerWiseDummyOptimizer(torch.optim.Optimizer):
    """
    For Layer-wise optimizers such as GaLoRE optimizer, the optimization
    step is already done through the post gradient hooks. Therefore
    the trick is to create a dummy optimizer that can take arbitrary
    args and kwargs and return a no-op during training.

    Initial idea from @hiyouga in LLaMA-Factory:
    https://github.com/hiyouga/LLaMA-Factory/commit/8664262cde3919e10eaecbd66e8c5d356856362e#diff-ebe08ab14496dfb9e06075f0fdd36799ef6d1535cc4dd4715b74c4e3e06fe3ba
    """

    def __init__(self, optimizer_dict=None, *args, **kwargs):
        dummy_tensor = torch.randn(1, 1)
        self.optimizer_dict = optimizer_dict
        super().__init__([dummy_tensor], {"lr": kwargs.get("lr", 1e-03)})

    def zero_grad(self, set_to_none: bool = True) -> None:
        pass

    def step(self, closure=None) -> Optional[float]:


SUMMARY:
**LayerWiseDummyOptimizer Class Summary**

The `LayerWiseDummyOptimizer` class is a custom optimization wrapper designed for Layer-wise optimizers like GaLoRE optimizer. Its primary purpose is to create a dummy optimizer that takes arbitrary arguments and returns no-op during training.

**Key Implementation Details:**

* Initializes with an optional `optimizer_dict`, a random tensor (`torch.randn(1, 1)`) and sets the learning rate using `kwargs.get("lr", 1e-03)`.
* Provides dummy implementations for `zero_grad` and `step` methods.

**Notable Dependencies:**

* Uses PyTorch library (`torch.optim.Optimizer`)
* References the `torch.randn` function
* Note that it inherits from `torch.optim.Optimizer`, which implements standard optimization algorithms, but uses a dummy implementation to achieve its specific purpose.


key: chroma:document
string_value:
CODE:

class LayerWiseDummyScheduler(LRScheduler):
    """
    For Layer-wise optimizers such as GaLoRE optimizer, the optimization and scheduling step
    are already done through the post gradient hooks. Therefore
    the trick is to create a dummy scheduler that can take arbitrary
    args and kwargs and return a no-op during training.
    """

    def __init__(self, *args, **kwargs):
        self.default_lr = kwargs["lr"]
        optimizer = LayerWiseDummyOptimizer(**kwargs)
        last_epoch = -1
        verbose = False
        super().__init__(optimizer, last_epoch, verbose)

    def get_lr(self):
        # default value
        lrs = [self.default_lr]

        # we take each lr in the parameters if they exist, assumes the optimizer to be the `LayerWiseDummyOptimizer`
        if self.optimizer is not None:
            param_wise_lrs = [
                [group["lr"] for group in optim.param_groups] for optim in self.optimizer.optimizer_dict.values()
            ]
            lrs = list(chain(*param_wise_lrs))

        return lrs

    def _get_closed_form_lr(self):


SUMMARY:
**LayerWiseDummyScheduler Class Summary**

*   **Purpose and Functionality:** The LayerWiseDummyScheduler is a mock scheduler that implements the LRScheduler interface. Its primary function is to return no-op operations during training. It acts as a dummy scheduler for optimizers like GaLoRE.
*   **Key Implementation Details:**
    *   `get_lr` method:
        *   Returns the initial learning rate (`self.default_lr`)
        *   If `self.optimizer` exists, it retrieves parameter-wise learning rates from the optimizer
    *   `_get_closed_form_lr` method (not implemented):
        *   Not available (returns `None` or an error message)

**Notable Dependencies:**

*   `LayerWiseDummyOptimizer`: The specific optimizer instance that extends this scheduler.

**External Dependencies:**

*   None

**Inherited/Implemented Interfaces:**

*   LRScheduler interface


key: chroma:document
string_value:
CODE:

class TransposeType(ExplicitEnum):
    """
    Possible ...
    """

    NO = "no"
    SIMPLE = "simple"
    CONV1D = "conv1d"


SUMMARY:
**Summary:**

The `TransposeType` class defines an enumeration with four possible values:

* `NO`: No transpose operation
* `SIMPLE`: Simple transpose (likely referring to spatial or batch dimension swapping)
* `CONV1D`: Transpose for 1D convolutional layers
* `CONV2D`: Transpose for 2D convolutional layers

The primary purpose of this enumeration is to provide a way to specify the type of transpose operation when working with neural networks, likely for model architecture or layer configurations.

**Key Implementation Details:**

* No specific algorithmic implementation details are present in this code snippet.
* The emphasis is on providing a clear and concise representation of possible transpose types.

**Notable Dependencies or Requirements:**

* None explicitly mentioned; however, the context suggests an environment with `ExplicitEnum` support, likely in a .NET or C#-based framework.


key: chroma:document
string_value:
CODE:

def convert_tf_weight_name_to_pt_weight_name(
    tf_name, start_prefix_to_remove="", tf_weight_shape=None, name_scope=None
):
    """
    Convert a TF 2.0 model variable name in a pytorch model weight name.

    Conventions for TF2.0 scopes -> PyTorch attribute names conversions:

        - '$1___$2' is replaced by $2 (can be used to duplicate or remove layers in TF2.0 vs PyTorch)
        - '_._' is replaced by a new level separation (can be used to convert TF2.0 lists in PyTorch nn.ModulesList)

    return tuple with:

        - pytorch model weight name
        - transpose: `TransposeType` member indicating whether and how TF2.0 and PyTorch weights matrices should be
          transposed with regards to each other
    """
    if name_scope is not None:
        if not tf_name.startswith(name_scope) and "final_logits_bias" not in tf_name:
            raise ValueError(
                f"Weight name {tf_name} does not start with name_scope {name_scope}. This is an internal error "
                "in Transformers, so (unless you were doing something really evil) please open an issue to report it!"
            )
        tf_name = tf_name[len(name_scope) :]
        tf_name = tf_name.lstrip("/")
    tf_name = tf_name.replace(":0", "")  # device ids
    tf_name = re.sub(
        r"/[^/]*___([^/]*)/", r"/\1/", tf_name
    )  # '$1___$2' is replaced by $2 (can be used to duplicate or remove layers in TF2.0 vs PyTorch)
    tf_name = tf_name.replace(
        "_._", "/"
    )  # '_._' is replaced by a level separation (can be used to convert TF2.0 lists in PyTorch nn.ModulesList)
    tf_name = re.sub(r"//+", "/", tf_name)  # Remove empty levels at the end
    tf_name = tf_name.split("/")  # Convert from TF2.0 '/' separators to PyTorch '.' separators
    # Some weights have a single name without "/" such as final_logits_bias in BART
    if len(tf_name) > 1:
        tf_name = tf_name[1:]  # Remove level zero

    tf_weight_shape = list(tf_weight_shape)

    # When should we transpose the weights
    if tf_name[-1] == "kernel" and tf_weight_shape is not None and len(tf_weight_shape) == 4:
        transpose = TransposeType.CONV2D
    elif tf_name[-1] == "kernel" and tf_weight_shape is not None and len(tf_weight_shape) == 3:
        transpose = TransposeType.CONV1D
    elif bool(
        tf_name[-1] in ["kernel", "pointwise_kernel", "depthwise_kernel"]
        or "emb_projs" in tf_name
        or "out_projs" in tf_name
    ):
        transpose = TransposeType.SIMPLE
    else:
        transpose = TransposeType.NO

    # Convert standard TF2.0 names in PyTorch names
    if tf_name[-1] == "kernel" or tf_name[-1] == "embeddings" or tf_name[-1] == "gamma":
        tf_name[-1] = "weight"
    if tf_name[-1] == "beta":
        tf_name[-1] = "bias"

    # The SeparableConv1D TF layer contains two weights that are translated to PyTorch Conv1D here
    if tf_name[-1] == "pointwise_kernel" or tf_name[-1] == "depthwise_kernel":
        tf_name[-1] = tf_name[-1].replace("_kernel", ".weight")

    # Remove prefix if needed
    tf_name = ".".join(tf_name)
    if start_prefix_to_remove:
        tf_name = tf_name.replace(start_prefix_to_remove, "", 1)



SUMMARY:
**Function Summary: `convert_tf_weight_name_to_pt_weight_name`**

This function converts a TF 2.0 model variable name to a PyTorch model weight name.

**Main Purpose and Functionality:**
The function takes a TF name, an optional parameter to remove from the start of the name, and optional parameters for weight shape and scope.
It returns a tuple containing:
1. The converted PyTorch model weight name
2. A `TransposeType` member indicating whether weights should be transposed between TF 2.0 and PyTorch

**Key Implementation Details:**

* Replaces special names like '$1___$2' with $2
* Converts '/' separators to '.'
* Transposes kernel weights if necessary, based on weight shape
* Standardizes weights as 'weight', biases as 'bias'

**Notable Dependencies or Requirements:**

* `re` module for regular expression replacements
* `TransposeType` member for transpose operation

* `name_scope` parameter assumes an internal error handling mechanism


key: chroma:document
string_value:
CODE:

def apply_transpose(transpose: TransposeType, weight, match_shape=None, pt_to_tf=True):
    """
    Apply a transpose to some weight then tries to reshape the weight to the same shape as a given shape, all in a
    framework agnostic way.
    """
    if transpose is TransposeType.CONV2D:
        # Conv2D weight:
        #    PT: (num_out_channel, num_in_channel, kernel[0], kernel[1])
        # -> TF: (kernel[0], kernel[1], num_in_channel, num_out_channel)
        axes = (2, 3, 1, 0) if pt_to_tf else (3, 2, 0, 1)
        weight = transpose_func(weight, axes=axes)
    elif transpose is TransposeType.CONV1D:
        # Conv1D weight:
        #    PT: (num_out_channel, num_in_channel, kernel)
        # -> TF: (kernel, num_in_channel, num_out_channel)
        weight = transpose_func(weight, axes=(2, 1, 0))
    elif transpose is TransposeType.SIMPLE:
        weight = transpose_func(weight)

    if match_shape is None:
        return weight

    if len(match_shape) < len(weight.shape):
        weight = squeeze(weight)
    elif len(match_shape) > len(weight.shape):
        weight = expand_dims(weight, axis=0)

    if list(match_shape) != list(weight.shape):
        try:
            weight = reshape(weight, match_shape)
        except AssertionError as e:
            e.args += (match_shape, match_shape)
            raise e



SUMMARY:
**Function Summary: `apply_transpose`**

**Purpose:** Apply transpose to a weight tensor and optionally reshape it to conform to a given target shape.

**Functionality:**

1. Applies transpose to the input weight based on the specified `transpose` type (`CONV2D`, `CONV1D`, or `SIMPLE`).
2. If a match shape is provided, resizes the weight tensor using:
	* `squeeze` if the source and target shapes have fewer dimensions.
	* `expand_dims` if the source and target shapes have more dimensions.
3. Attempts to reshape the weighted tensor to conform to the target shape using `reshape`. Fails with an assertion error if the shapes do not match.

**Notable Details:**

* Utilizes `transpose_func` for transpose operations (Conv2D) and a generic version `(weight, axes=axes)` for simple transposes.
* Applies transpose type-specific transformations:
	+ Conv2D: flips dimensions (PT -> TF).
	+ Conv1D: rearranges dimensions (PT -> TF).
* Supports match-shaped resizing using `squeeze` and `expand_dims`.
* Raises an assertion error if the target shape does not conform to the source tensor.

**Dependencies/Requirements:**

* `TransposeType`: enumeration of transpose types (`CONV2D`, `CONV1D`, or `SIMPLE`)
* `transpose_func`: a function implementing transpose operations
* `squeeze` and `expand_dims` functions for resizing tensors
* `reshape` function with support for conditional exception handling

**External Interfaces:**

* `TransposeType`
* `transpose_func`
* `squeeze`
* `expand_dims`
* `reshape`


key: chroma:document
string_value:
CODE:

def load_pytorch_checkpoint_in_tf2_model(
    tf_model,
    pytorch_checkpoint_path,
    tf_inputs=None,
    allow_missing_keys=False,
    output_loading_info=False,
    _prefix=None,
    tf_to_pt_weight_rename=None,
):
    """Load pytorch checkpoints in a TF 2.0 model"""
    try:
        import tensorflow as tf  # noqa: F401
        import torch  # noqa: F401
        from safetensors.torch import load_file as safe_load_file  # noqa: F401
    except ImportError:
        logger.error(
            "Loading a PyTorch model in TensorFlow, requires both PyTorch and TensorFlow to be installed. Please see "
            "https://pytorch.org/ and https://www.tensorflow.org/install/ for installation instructions."
        )
        raise

    # Treats a single file as a collection of shards with 1 shard.
    if isinstance(pytorch_checkpoint_path, str):
        pytorch_checkpoint_path = [pytorch_checkpoint_path]

    # Loads all shards into a single state dictionary
    pt_state_dict = {}
    for path in pytorch_checkpoint_path:
        pt_path = os.path.abspath(path)
        logger.info(f"Loading PyTorch weights from {pt_path}")
        if pt_path.endswith(".safetensors"):
            state_dict = safe_load_file(pt_path)
        else:
            weights_only_kwarg = {"weights_only": True}
            state_dict = torch.load(pt_path, map_location="cpu", **weights_only_kwarg)

        pt_state_dict.update(state_dict)

    logger.info(f"PyTorch checkpoint contains {sum(t.numel() for t in pt_state_dict.values()):,} parameters")

    return load_pytorch_weights_in_tf2_model(
        tf_model,
        pt_state_dict,
        tf_inputs=tf_inputs,
        allow_missing_keys=allow_missing_keys,
        output_loading_info=output_loading_info,
        _prefix=_prefix,
        tf_to_pt_weight_rename=tf_to_pt_weight_rename,


SUMMARY:
**Summary**

loads PyTorch checkpoints into a TensorFlow 2.0 model, handling state dictionary merging and weight loading.

**Main Purpose and Functionality**
```python
def load_pytorch_checkpoint_in_tf2_model(
    tf_model,
    pytorch_checkpoint_path,
    ...
):
```
This function loads a PyTorch checkpoint into a TF 2.0 model, treating a single file as a collection of shards with one shard. It handles loading weights from individual checkpoint files using `safe_load_file` and `torch.load`, and merges them into a unified state dictionary.

**Key Implementation Details**

* Loads `pytorch_checkpoint_path` using `os.path.abspath` to get the absolute path.
* Uses `safe_load_file` for `.safetensors` checkpoints and `torch.load` for other checkpoint files, with `weights_only=True` for optimal performance.
* Merges loaded state dictionaries into a single dictionary using `pt_state_dict.update(state_dict)`.

**Dependencies and Requirements**

* TensorFlow
* PyTorch
* Safetensors (library)
* Python logger (`logger.error`, `logger.info`)


key: chroma:document
string_value:
CODE:

def load_pytorch_model_in_tf2_model(tf_model, pt_model, tf_inputs=None, allow_missing_keys=False):
    """Load pytorch checkpoints in a TF 2.0 model"""
    pt_state_dict = pt_model.state_dict()

    return load_pytorch_weights_in_tf2_model(
        tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys


SUMMARY:
**Summary:**

`load_pytorch_model_in_tf2_model` Function:

Loads PyTorch models into TF 2.0 models.

### Main Purpose and Functionality:

loads PT_state_dict into TF_model using `load_pytorch_weights_in_tf2_model`.

### Key Implementation Details:

- Loads PyTorch state dictionary (`pt_state_dict`) from `tf_model`.
- Calls `load_pytorch_weights_in_tf2_model` with `tf_model`, `pt_state_dict`, and optional inputs.
  
### Notable Dependencies or Requirements:

- `load_pytorch_weights_in_tf2_model`
- `tf_model` (TF 2.0 model)
- `pt_model` (PyTorch model)
- External dependencies on specific methods and classes implied by the function signature.


key: chroma:document
string_value:
CODE:

def load_pytorch_weights_in_tf2_model(
    tf_model,
    pt_state_dict,
    tf_inputs=None,
    allow_missing_keys=False,
    output_loading_info=False,
    _prefix=None,
    tf_to_pt_weight_rename=None,
):
    """Load pytorch state_dict in a TF 2.0 model."""
    try:
        import tensorflow as tf  # noqa: F401
        import torch  # noqa: F401
    except ImportError:
        logger.error(
            "Loading a PyTorch model in TensorFlow, requires both PyTorch and TensorFlow to be installed. Please see "
            "https://pytorch.org/ and https://www.tensorflow.org/install/ for installation instructions."
        )
        raise

    # Numpy doesn't understand bfloat16, so upcast to a dtype that doesn't lose precision
    pt_state_dict = {
        k: v.numpy() if v.dtype != torch.bfloat16 else v.float().numpy() for k, v in pt_state_dict.items()
    }
    return load_pytorch_state_dict_in_tf2_model(
        tf_model,
        pt_state_dict,
        tf_inputs=tf_inputs,
        allow_missing_keys=allow_missing_keys,
        output_loading_info=output_loading_info,
        _prefix=_prefix,
        tf_to_pt_weight_rename=tf_to_pt_weight_rename,


SUMMARY:
**Summary**

`load_pytorch_weights_in_tf2_model` loads PyTorch state dictionary into a TensorFlow 2.0 model.

**Key Implementation Details**

* Converts PyTorch weights to compatible tensors using `pt_state_dict.item()`.
* Calls `load_pytorch_state_dict_in_tf2_model` with adapted input parameters.
* Handles potential missing keys in the PyTorch state dictionary with configurable behavior.

**Notable Dependencies or Requirements**

* `tensorflow` (TF 2.0) - loaded using `import tensorflow as tf`
* `torch` - loaded using `import torch`
* `numpy` (implied by tensor upcasting)

**Primary Purpose**

Load and integrate PyTorch state dictionary into a TensorFlow 2.0 model, handling potential data type compatibility issues.


key: chroma:document
string_value:
CODE:

def _log_key_warnings(missing_keys, unexpected_keys, mismatched_keys, class_name):
    if len(unexpected_keys) > 0:
        logger.warning(
            "Some weights of the PyTorch model were not used when initializing the TF 2.0 model"
            f" {class_name}: {unexpected_keys}\n- This IS expected if you are initializing"
            f" {class_name} from a PyTorch model trained on another task or with another architecture"
            " (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).\n- This IS"
            f" NOT expected if you are initializing {class_name} from a PyTorch model that you expect"
            " to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a"
            " BertForSequenceClassification model)."
        )
    else:
        logger.warning(f"All PyTorch model weights were used when initializing {class_name}.\n")
    if len(missing_keys) > 0:
        logger.warning(
            f"Some weights or buffers of the TF 2.0 model {class_name} were not initialized from the"
            f" PyTorch model and are newly initialized: {missing_keys}\nYou should probably TRAIN this model on a"
            " down-stream task to be able to use it for predictions and inference."
        )
    else:
        logger.warning(
            f"All the weights of {class_name} were initialized from the PyTorch model.\n"
            "If your task is similar to the task the model of the checkpoint was trained on, "
            f"you can already use {class_name} for predictions without further training."
        )

    if len(mismatched_keys) > 0:
        mismatched_warning = "\n".join(
            [
                f"- {key}: found shape {shape1} in the checkpoint and {shape2} in the model instantiated"
                for key, shape1, shape2 in mismatched_keys
            ]
        )
        logger.warning(
            f"Some weights of {class_name} were not initialized from the model checkpoint"
            f" are newly initialized because the shapes did not"
            f" match:\n{mismatched_warning}\nYou should probably TRAIN this model on a down-stream task to be able"
            " to use it for predictions and inference."


SUMMARY:
**Function Summary: `_log_key_warnings`**

### Purpose and Functionality

This function logs warnings related to PyTorch model initialization in the TF 2.0 framework. It checks for:

* Unused weights (unexpected keys) when initializing from a PyTorch model trained on another task or architecture.
* Missing weights or buffers not initialized from the PyTorch model.
* Mismatched key shapes between the checkpoint and model instantiated.

### Key Implementation Details

* Logs warnings using the `logger` object with contextual messages for each warning type (unused, missing, mismatched keys).
* Uses list comprehensions to generate formatted strings for display in the warning logs.

### Notable Dependencies and Requirements

* `pytorch`: The PyTorch library, which is assumed to be used as a checkpoint or model source.
* `tf`: The TensorFlow 2.0 framework, which is used to initialize models.
* `logger` object: A logging mechanism for displaying warnings about key mismatches.

### Method Signatures and Interfaces

* `_log_key_warnings(missing_keys, unexpected_keys, mismatched_keys, class_name)`:
	+ Parameters:
	+ `missing_keys`: List of missing weights or buffers.
	+ `unexpected_keys`: List of unused weights.
	+ `mismatched_keys`: List of mismatched key shapes between the checkpoint and model instantiated.
	+ `class_name`: The name of the class that is being initialized.

### Interface Inheritance

This function does not inherit any interfaces, but it uses built-in Python lists to store data.


key: chroma:document
string_value:
CODE:

def load_pytorch_state_dict_in_tf2_model(
    tf_model,
    pt_state_dict,
    tf_inputs=None,
    allow_missing_keys=False,
    output_loading_info=False,
    _prefix=None,
    tf_to_pt_weight_rename=None,
    ignore_mismatched_sizes=False,
    skip_logger_warnings=False,
):
    """Load a pytorch state_dict in a TF 2.0 model. pt_state_dict can be either an actual dict or a lazy-loading
    safetensors archive created with the safe_open() function."""
    import tensorflow as tf

    if tf_inputs is None:
        tf_inputs = tf_model.dummy_inputs

    if _prefix is None:
        _prefix = ""
    if tf_inputs:
        with tf.name_scope(_prefix):
            tf_model(tf_inputs, training=False)  # Make sure model is built
    # Convert old format to new format if needed from a PyTorch state_dict
    tf_keys_to_pt_keys = {}
    for key in pt_state_dict.keys():
        new_key = None
        if "gamma" in key:
            new_key = key.replace("gamma", "weight")
        if "beta" in key:
            new_key = key.replace("beta", "bias")
        if "running_var" in key:
            new_key = key.replace("running_var", "moving_variance")
        if "running_mean" in key:
            new_key = key.replace("running_mean", "moving_mean")

        # New `weight_norm` from https://github.com/huggingface/transformers/pull/24030
        key_components = key.split(".")
        name = None
        if key_components[-3::2] == ["parametrizations", "original0"]:
            name = key_components[-2] + "_g"
        elif key_components[-3::2] == ["parametrizations", "original1"]:
            name = key_components[-2] + "_v"
        if name is not None:
            key_components = key_components[:-3] + [name]
            new_key = ".".join(key_components)

        if new_key is None:
            new_key = key
        tf_keys_to_pt_keys[new_key] = key

    # Matt: All TF models store the actual model stem in a MainLayer class, including the base model.
    # In PT, the derived models (with heads) use the base model class as the stem instead,
    # and there is no MainLayer class. This means that TF base classes have one
    # extra layer in their weight names, corresponding to the MainLayer class. This code block compensates for that.
    start_prefix_to_remove = ""
    if not any(s.startswith(tf_model.base_model_prefix) for s in tf_keys_to_pt_keys.keys()):
        start_prefix_to_remove = tf_model.base_model_prefix + "."

    symbolic_weights = tf_model.trainable_weights + tf_model.non_trainable_weights
    tf_loaded_numel = 0
    all_pytorch_weights = set(tf_keys_to_pt_keys.keys())
    missing_keys = []
    mismatched_keys = []
    is_safetensor_archive = hasattr(pt_state_dict, "get_tensor")
    for symbolic_weight in symbolic_weights:
        sw_name = symbolic_weight.name
        name, transpose = convert_tf_weight_name_to_pt_weight_name(
            sw_name,
            start_prefix_to_remove=start_prefix_to_remove,
            tf_weight_shape=symbolic_weight.shape,
            name_scope=_prefix,
        )
        if tf_to_pt_weight_rename is not None:
            aliases = tf_to_pt_weight_rename(name)  # Is a tuple to account for possible name aliasing
            for alias in aliases:  # The aliases are in priority order, take the first one that matches
                if alias in tf_keys_to_pt_keys:
                    name = alias
                    break
            else:
                # If none of the aliases match, just use the first one (it'll be reported as missing)
                name = aliases[0]

        # Find associated numpy array in pytorch model state dict
        if name not in tf_keys_to_pt_keys:
            if allow_missing_keys:
                missing_keys.append(name)
                continue
            elif tf_model._keys_to_ignore_on_load_missing is not None:
                # authorized missing keys don't have to be loaded
                if any(re.search(pat, name) is not None for pat in tf_model._keys_to_ignore_on_load_missing):
                    continue
            raise AttributeError(f"{name} not found in PyTorch model")
        state_dict_name = tf_keys_to_pt_keys[name]
        if is_safetensor_archive:
            array = pt_state_dict.get_tensor(state_dict_name)
        else:
            array = pt_state_dict[state_dict_name]
        try:
            array = apply_transpose(transpose, array, symbolic_weight.shape)
        except tf.errors.InvalidArgumentError as e:
            if not ignore_mismatched_sizes:
                error_msg = str(e)
                error_msg += (
                    "\n\tYou may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method."
                )
                raise tf.errors.InvalidArgumentError(error_msg)
            else:
                mismatched_keys.append((name, array.shape, symbolic_weight.shape))
                continue

        tf_loaded_numel += tensor_size(array)

        symbolic_weight.assign(tf.cast(array, symbolic_weight.dtype))
        del array  # Immediately free memory to keep peak usage as low as possible
        all_pytorch_weights.discard(name)

    logger.info(f"Loaded {tf_loaded_numel:,} parameters in the TF 2.0 model.")

    unexpected_keys = list(all_pytorch_weights)

    if tf_model._keys_to_ignore_on_load_missing is not None:
        for pat in tf_model._keys_to_ignore_on_load_missing:
            missing_keys = [k for k in missing_keys if re.search(pat, k) is None]
    if tf_model._keys_to_ignore_on_load_unexpected is not None:
        for pat in tf_model._keys_to_ignore_on_load_unexpected:
            unexpected_keys = [k for k in unexpected_keys if re.search(pat, k) is None]
    if not skip_logger_warnings:
        _log_key_warnings(missing_keys, unexpected_keys, mismatched_keys, class_name=tf_model.__class__.__name__)

    if output_loading_info:
        loading_info = {
            "missing_keys": missing_keys,
            "unexpected_keys": unexpected_keys,
            "mismatched_keys": mismatched_keys,
        }
        return tf_model, loading_info



SUMMARY:
**Summary: Load PyTorch State Dict in TF 2.0 Model**

This function loads a PyTorch state dictionary into a TensorFlow 2.0 model.

**Purpose and Functionality**

The primary purpose is to map PyTorch weights to their corresponding TensorFlow counterparts, handling prefixes, aliases, and missing keys. The function converts old format to new format if needed.

**Implementation Details**

1. ** renaming keys**: The code maps TF keys to PT keys using predefined renaming rules.
2. **convert_tf_weight_name_to_pt_weightName**: This function transforms weight names from TF to PT format.
3. **load_numel**: Calculates the total number of loaded parameters.
4. **assign_weights**: Assigns PyTorch weights to corresponding TensorFlow weights.

**Key Implementation Details**

* `tf_inputs`: Dummy inputs for the model.
* `_prefix`: Name scope for logging and naming consistency.
* `tf_keys_to_pt_keys`: A dictionary mapping TF keys to PT keys.
* `apply_transpose`: Transposes the array based on the specified transpose order.
* `tensor_size`: Calculates the size of a tensor.

**Dependencies**

* TensorFlow 2.0
* PyTorch state dictionary (safetensors archive or regular dict)

**Requirements**

* `.trainable_weights` and `.non_trainable_weights` attributes of the model.
* `_keys_to_ignore_on_load_missing` and `_keys_to_ignore_on_load_unexpected` attributes of the model.

Note that specific implementation details, such as the naming rules and aliasing mechanism, are not included in this summary.


key: chroma:document
string_value:
CODE:

def load_sharded_pytorch_safetensors_in_tf2_model(
    tf_model,
    safetensors_shards,
    tf_inputs=None,
    allow_missing_keys=False,
    output_loading_info=False,
    _prefix=None,
    tf_to_pt_weight_rename=None,
    ignore_mismatched_sizes=False,
):
    all_loading_infos = []
    for shard in safetensors_shards:
        with safe_open(shard, framework="tf") as safetensors_archive:
            tf_model, loading_info = load_pytorch_state_dict_in_tf2_model(
                tf_model,
                safetensors_archive,
                tf_inputs=tf_inputs,
                allow_missing_keys=allow_missing_keys,
                output_loading_info=True,
                _prefix=_prefix,
                tf_to_pt_weight_rename=tf_to_pt_weight_rename,
                ignore_mismatched_sizes=ignore_mismatched_sizes,
                skip_logger_warnings=True,  # We will emit merged warnings at the end
            )
        all_loading_infos.append(loading_info)
    # Now we just need to merge the loading info
    # Keys are missing only if they're missing in *every* shard
    missing_keys = sorted(set.intersection(*[set(info["missing_keys"]) for info in all_loading_infos]))
    # Keys are unexpected/mismatched if they're unexpected/mismatched in *any* shard
    unexpected_keys = sum([info["unexpected_keys"] for info in all_loading_infos], [])
    mismatched_keys = sum([info["mismatched_keys"] for info in all_loading_infos], [])

    _log_key_warnings(missing_keys, unexpected_keys, mismatched_keys, class_name=tf_model.__class__.__name__)

    if output_loading_info:
        loading_info = {
            "missing_keys": missing_keys,
            "unexpected_keys": unexpected_keys,
            "mismatched_keys": mismatched_keys,
        }
        return tf_model, loading_info



SUMMARY:
**Summary**

 loads PyTorch Safetensors in TF2 model and merges loading information.

**Main Functionality:**

* Loads Safetensors archives for each shard in a sharded dataset
* Merges loading information from all shards to identify missing, unexpected, and mismatched keys.


key: chroma:document
string_value:
CODE:

def load_tf2_checkpoint_in_pytorch_model(
    pt_model, tf_checkpoint_path, tf_inputs=None, allow_missing_keys=False, output_loading_info=False
):
    """
    Load TF 2.0 HDF5 checkpoint in a PyTorch model We use HDF5 to easily do transfer learning (see
    https://github.com/tensorflow/tensorflow/blob/ee16fcac960ae660e0e4496658a366e2f745e1f0/tensorflow/python/keras/engine/network.py#L1352-L1357).
    """
    try:
        import tensorflow as tf  # noqa: F401
        import torch  # noqa: F401
    except ImportError:
        logger.error(
            "Loading a TensorFlow model in PyTorch, requires both PyTorch and TensorFlow to be installed. Please see "
            "https://pytorch.org/ and https://www.tensorflow.org/install/ for installation instructions."
        )
        raise

    import transformers

    from .modeling_tf_utils import load_tf_weights

    logger.info(f"Loading TensorFlow weights from {tf_checkpoint_path}")

    # Instantiate and load the associated TF 2.0 model
    tf_model_class_name = "TF" + pt_model.__class__.__name__  # Add "TF" at the beginning
    tf_model_class = getattr(transformers, tf_model_class_name)
    tf_model = tf_model_class(pt_model.config)

    if tf_inputs is None:
        tf_inputs = tf_model.dummy_inputs

    if tf_inputs is not None:
        tf_model(tf_inputs, training=False)  # Make sure model is built

    load_tf_weights(tf_model, tf_checkpoint_path)

    return load_tf2_model_in_pytorch_model(
        pt_model, tf_model, allow_missing_keys=allow_missing_keys, output_loading_info=output_loading_info


SUMMARY:
**Summary**
------------

`load_tf2_checkpoint_in_pytorch_model`: Load a TensorFlow 2.0 checkpoint into a PyTorch model.

### Purpose and Functionality
Loads the specified TensorFlow 2.0 checkpoint into a PyTorch model, utilizing transfer learning through HDF5 files.

### Implementation Details

*   Uses `transformers` library to instantiate the corresponding TF 2.0 model class.
*   Loads the checkpoint into the instantiated model using `load_tf_weights`.
*   Optionally loads pre-defined inputs for the model by calling `tf_model(tf_inputs, training=False)`.

### Notable Dependencies and Requirements

*   TensorFlow (`tf`)
*   PyTorch
*   Transformers library (imported)
*   `logger` object (assumed external dependency)

**Key Signatures:**

*   `load_tf_weights(tf_model, tf_checkpoint_path)`
*   `tf_model_class.pt_model.config`
*   `load_tf2_model_in_pytorch_model(pt_model, tf_model, ..., allow_missing_keys=allow_missing_keys, output_loading_info=output_loading_info)`


key: chroma:document
string_value:
CODE:

def load_tf2_model_in_pytorch_model(pt_model, tf_model, allow_missing_keys=False, output_loading_info=False):
    """Load TF 2.0 model in a pytorch model"""
    weights = tf_model.weights

    return load_tf2_weights_in_pytorch_model(
        pt_model, weights, allow_missing_keys=allow_missing_keys, output_loading_info=output_loading_info


SUMMARY:
**Summary**

Loads a TF 2.0 model into a PyTorch model.

### Key Implementation Details

* Utilizes `weights` from the input TF 2.0 model
* Calls `load_tf2_weights_in_pytorch_model` with PyTorch model, weights, and optional flags for missing keys and loading info.

### Notable Dependencies/Requirements

* `tf_model.weights`
* Input `pt_model` (PyTorch model)
* Optional input parameters:
	+ `allow_missing_keys=False`: Flag to specify handling of missing keys
	+ `output_loading_info=False`: Flag to specify outputting loading information


key: chroma:document
string_value:
CODE:

def load_tf2_weights_in_pytorch_model(pt_model, tf_weights, allow_missing_keys=False, output_loading_info=False):
    """Load TF2.0 symbolic weights in a PyTorch model"""
    try:
        import tensorflow as tf  # noqa: F401
        import torch  # noqa: F401
    except ImportError:
        logger.error(
            "Loading a TensorFlow model in PyTorch, requires both PyTorch and TensorFlow to be installed. Please see "
            "https://pytorch.org/ and https://www.tensorflow.org/install/ for installation instructions."
        )
        raise

    tf_state_dict = {tf_weight.name: tf_weight.numpy() for tf_weight in tf_weights}
    return load_tf2_state_dict_in_pytorch_model(
        pt_model, tf_state_dict, allow_missing_keys=allow_missing_keys, output_loading_info=output_loading_info


SUMMARY:
**load_tf2_weights_in_pytorch_model Function Summary**

**Main Purpose and Functionality**
---------------------------

This function loads TensorFlow 2.0 symbolic weights into a PyTorch model.

* It takes four inputs: `pt_model` (the PyTorch model), `tf_weights` (TensorFlow 2.0 weights), `allow_missing_keys`, and `output_loading_info`.
* The function attempts to fetch the necessary libraries (PyTorch and TensorFlow) and raises an error if neither is installed.

**Key Implementation Details**
-------------------------

* This function uses dictionary comprehension to create a `tf_state_dict` with TF weights' names as keys and their numpy representations as values.
* It then calls another function, `load_tf2_state_dict_in_pytorch_model`, to load the state dict into the PyTorch model.

**Notable Dependencies or Requirements**
--------------------------------------

* TensorFlow (TF) library
* PyTorch library

**Method Signatures and Interfaces**
-----------------------------------

* The function takes four arguments: `pt_model`, `tf_weights`, `allow_missing_keys`, and `output_loading_info`.
* It returns the loaded state dict in the PyTorch model.


key: chroma:document
string_value:
CODE:

def load_tf2_state_dict_in_pytorch_model(pt_model, tf_state_dict, allow_missing_keys=False, output_loading_info=False):
    import torch

    new_pt_params_dict = {}
    current_pt_params_dict = dict(pt_model.named_parameters())

    # Make sure we are able to load PyTorch base models as well as derived models (with heads)
    # TF models always have a prefix, some of PyTorch models (base ones) don't
    start_prefix_to_remove = ""
    if not any(s.startswith(pt_model.base_model_prefix) for s in current_pt_params_dict.keys()):
        start_prefix_to_remove = pt_model.base_model_prefix + "."

    # Build a map from potential PyTorch weight names to TF 2.0 Variables
    tf_weights_map = {}
    for name, tf_weight in tf_state_dict.items():
        pt_name, transpose = convert_tf_weight_name_to_pt_weight_name(
            name, start_prefix_to_remove=start_prefix_to_remove, tf_weight_shape=tf_weight.shape
        )
        tf_weights_map[pt_name] = (tf_weight, transpose)

    all_tf_weights = set(tf_weights_map.keys())
    loaded_pt_weights_data_ptr = {}
    missing_keys_pt = []
    for pt_weight_name, pt_weight in current_pt_params_dict.items():
        # Handle PyTorch shared weight ()not duplicated in TF 2.0
        if pt_weight.data_ptr() in loaded_pt_weights_data_ptr:
            new_pt_params_dict[pt_weight_name] = loaded_pt_weights_data_ptr[pt_weight.data_ptr()]
            continue

        pt_weight_name_to_check = pt_weight_name
        # New `weight_norm` from https://github.com/huggingface/transformers/pull/24030
        key_components = pt_weight_name.split(".")
        name = None
        if key_components[-3::2] == ["parametrizations", "original0"]:
            name = key_components[-2] + "_g"
        elif key_components[-3::2] == ["parametrizations", "original1"]:
            name = key_components[-2] + "_v"
        if name is not None:
            key_components = key_components[:-3] + [name]
            pt_weight_name_to_check = ".".join(key_components)

        # Find associated numpy array in pytorch model state dict
        if pt_weight_name_to_check not in tf_weights_map:
            if allow_missing_keys:
                missing_keys_pt.append(pt_weight_name)
                continue

            raise AttributeError(f"{pt_weight_name} not found in TF 2.0 model")

        array, transpose = tf_weights_map[pt_weight_name_to_check]

        array = apply_transpose(transpose, array, pt_weight.shape, pt_to_tf=False)

        if numpy.isscalar(array):
            array = numpy.array(array)
        if not is_torch_tensor(array) and not is_numpy_array(array):
            array = array.numpy()
        if is_numpy_array(array):
            # Convert to torch tensor
            array = torch.from_numpy(array)

        new_pt_params_dict[pt_weight_name] = array
        loaded_pt_weights_data_ptr[pt_weight.data_ptr()] = array
        all_tf_weights.discard(pt_weight_name)

    missing_keys, unexpected_keys = pt_model.load_state_dict(new_pt_params_dict, strict=False)
    missing_keys += missing_keys_pt

    # Some models may have keys that are not in the state by design, removing them before needlessly warning
    # the user.
    if pt_model._keys_to_ignore_on_load_missing is not None:
        for pat in pt_model._keys_to_ignore_on_load_missing:
            missing_keys = [k for k in missing_keys if re.search(pat, k) is None]

    if pt_model._keys_to_ignore_on_load_unexpected is not None:
        for pat in pt_model._keys_to_ignore_on_load_unexpected:
            unexpected_keys = [k for k in unexpected_keys if re.search(pat, k) is None]

    if len(unexpected_keys) > 0:
        logger.warning(
            "Some weights of the TF 2.0 model were not used when initializing the PyTorch model"
            f" {pt_model.__class__.__name__}: {unexpected_keys}\n- This IS expected if you are initializing"
            f" {pt_model.__class__.__name__} from a TF 2.0 model trained on another task or with another architecture"
            " (e.g. initializing a BertForSequenceClassification model from a TFBertForPreTraining model).\n- This IS"
            f" NOT expected if you are initializing {pt_model.__class__.__name__} from a TF 2.0 model that you expect"
            " to be exactly identical (e.g. initializing a BertForSequenceClassification model from a"
            " TFBertForSequenceClassification model)."
        )
    else:
        logger.warning(f"All TF 2.0 model weights were used when initializing {pt_model.__class__.__name__}.\n")
    if len(missing_keys) > 0:
        logger.warning(
            f"Some weights of {pt_model.__class__.__name__} were not initialized from the TF 2.0 model and are newly"
            f" initialized: {missing_keys}\nYou should probably TRAIN this model on a down-stream task to be able to"
            " use it for predictions and inference."
        )
    else:
        logger.warning(
            f"All the weights of {pt_model.__class__.__name__} were initialized from the TF 2.0 model.\n"
            "If your task is similar to the task the model of the checkpoint was trained on, "
            f"you can already use {pt_model.__class__.__name__} for predictions without further training."
        )

    logger.info(f"Weights or buffers not loaded from TF 2.0 model: {all_tf_weights}")

    if output_loading_info:
        loading_info = {"missing_keys": missing_keys, "unexpected_keys": unexpected_keys}
        return pt_model, loading_info



SUMMARY:
**Function Summary:**
`load_tf2_state_dict_in_pytorch_model`

This function loads a pre-trained PyTorch model by transferring weights from a TensorFlow 2.0 model.

**Purpose and Functionality:**

* Loads PyTorch model parameters (`pt_model`) based on a provided TensorFlow 2.0 state dictionary (`tf_state_dict`)
* Handles missing keys, duplicate weights, and unexpected key patterns between the two models
* Logs loading information, including missing keys, unexpected keys, and loaded weights
* Returns the updated `pt_model` with loaded parameters

**Key Implementation Details:**

* Uses a mapping function (`convert_tf_weight_name_to_pt_weight_name`) to convert TF2.0 weight names to PyTorch-compatible names
* Applies transpose operations for compatible arrays between TF2.0 and PyTorch
* Handles cases where certain keys are ignored or replaced during loading

**Notable Dependencies:**

* `torch`
* `numpy` (imported implicitly)

**Interfaces and Inherited Interfaces:**

* `pytorch.nn.Module` (`pt_model`)
* `tensorflow.python.framework ops.Eval`
* `numpy.ndarray`


key: chroma:document
string_value:
CODE:

def parse_flag_from_env(key, default=False):
    try:
        value = os.environ[key]
    except KeyError:
        # KEY isn't set, default to `default`.
        _value = default
    else:
        # KEY is set, convert it to True or False.
        try:
            _value = strtobool(value)
        except ValueError:
            # More values are supported, but let's keep the message simple.
            raise ValueError(f"If set, {key} must be yes or no.")


SUMMARY:
**Summary:**

`parse_flag_from_env` function:

* Retrieves environment variable `key` and defaults to `default=False` if not present.
* Converts the value to a boolean (`True`/`False`) or raises a `ValueError` if the value is not in binary format.

**Key Implementation Details:**

* Uses `os.environ[key]` to access environment variable
* Employes `strtobool` function to convert non-boolean values to booleans

**Notable Dependencies and Requirements:**

* `os` module (for accessing environment variables)
* Internal dependency on `_value` variable to store the converted value


key: chroma:document
string_value:
CODE:

def parse_int_from_env(key, default=None):
    try:
        value = os.environ[key]
    except KeyError:
        _value = default
    else:
        try:
            _value = int(value)
        except ValueError:
            raise ValueError(f"If set, {key} must be a int.")


SUMMARY:
**Summary**

`parse_int_from_env`: Extracts an integer value from an environment variable, returning the parsed integer or a default if the variable is not present. If parsing fails due to non-integer value, raises a `ValueError`.

**Key Implementation Details**

* Uses `os.environ[key]` to retrieve environment variable value.
* Attempts to parse the value as an integer using `int(value)`.
* Fails with a `ValueError` if parsing fails.

**Notable Dependencies and Requirements**

* `os` module for accessing environment variables.


key: chroma:document
string_value:
CODE:

def get_device_count():
    import torch

    if is_torch_xpu_available():
        num_devices = torch.xpu.device_count()
    else:
        num_devices = torch.cuda.device_count()



SUMMARY:
**Code Summary**

`get_device_count()` function returns the total number of devices (Accelerators) available in the system. The implementation uses PyTorch's `xpu` and/or `cuda` modules to determine device count.

**Key Details:**

* Primary purpose: Returns the total number of devices.
* Implementation details:
	+ Uses `torch.xpu.device_count()` to query XPU devices (if available).
	+ Fallbacks to `torch.cuda.device_count()` if XPU is not available.
* Notable dependencies:
	+ PyTorch library (`import torch`)
	+ `is_torch_xpu_available()` function (not shown in this snippet)


key: chroma:document
string_value:
CODE:

def is_pt_tf_cross_test(test_case):
    """
    Decorator marking a test as a test that control interactions between PyTorch and TensorFlow.

    PT+TF tests are skipped by default and we can run only them by setting RUN_PT_TF_CROSS_TESTS environment variable
    to a truthy value and selecting the is_pt_tf_cross_test pytest mark.

    """
    if not _run_pt_tf_cross_tests or not is_torch_available() or not is_tf_available():
        return unittest.skip(reason="test is PT+TF test")(test_case)
    else:
        try:
            import pytest  # We don't need a hard dependency on pytest in the main library
        except ImportError:
            return test_case
        else:


SUMMARY:
**Summary**

`is_pt_tf_cross_test` Decorator:

*   Marks tests as control interactions between PyTorch and TensorFlow (PT+TF) tests.
*   Skips tests by default unless `RUN_PT_TF_CROSS_TESTS` environment variable is set to a truthy value.

**Key Implementation Details:**

*   Returns `unittest.skip` or sets `pytest.mark.is_pt_tf_cross_test` based on availability of PT+TF tests and underlying framework integrations.

**Notable Dependencies/Requirements:**

*   `_run_pt_tf_cross_tests`: a boolean flag indicating whether to run PT+TF cross-tests.
*   `is_torch_available()`: checks if PyTorch is available for use.
*   `is_tf_available()`: checks if TensorFlow is available for use.
*   `pytest` (optional): import statement included in case of pytest integration.


key: chroma:document
string_value:
CODE:

def is_pt_flax_cross_test(test_case):
    """
    Decorator marking a test as a test that control interactions between PyTorch and Flax

    PT+FLAX tests are skipped by default and we can run only them by setting RUN_PT_FLAX_CROSS_TESTS environment
    variable to a truthy value and selecting the is_pt_flax_cross_test pytest mark.

    """
    if not _run_pt_flax_cross_tests or not is_torch_available() or not is_flax_available():
        return unittest.skip(reason="test is PT+FLAX test")(test_case)
    else:
        try:
            import pytest  # We don't need a hard dependency on pytest in the main library
        except ImportError:
            return test_case
        else:


SUMMARY:
### Code Summary

**Function:** `is_pt_flax_cross_test`

**Purpose:** Decorator marking tests for interactions between PyTorch and Flax.

**Functionality:**

* Skips PT+FLAX tests by default unless environment variable `RUN_PT_FLAX_CROSS_TESTS` is set to truthy value.
* Returns skipped test with reason "test is PT+FLAX test" if dependencies are not met.
* Imports pytest only when running as main library, and marks test with `@pytest.mark.is_pt_flax_cross_test()` otherwise.

### Notable Dependencies:
- `_run_pt_flax_cross_tests`
- `is_torch_available()`
- `is_flax_available()`

### Key Algorithm:
- Conditional checks for dependencies and environment variable.
- Optional pytest import and mark.


key: chroma:document
string_value:
CODE:

def is_staging_test(test_case):
    """
    Decorator marking a test as a staging test.

    Those tests will run using the staging environment of huggingface.co instead of the real model hub.
    """
    if not _run_staging:
        return unittest.skip(reason="test is staging test")(test_case)
    else:
        try:
            import pytest  # We don't need a hard dependency on pytest in the main library
        except ImportError:
            return test_case
        else:


SUMMARY:
**Function Summary: `is_staging_test` Decorator**

### Purpose and Functionality

The `is_staging_test` function serves as a decorator to mark test cases as staging tests. It substitutes the real model hub with the huggingface.co staging environment.

### Key Implementation Details

*   Uses the `unittest.skip` function with a custom reason if `_run_staging` is `False`.
*   Attempts to import `pytest` when `_run_staging` is `True`, and sets the decorated test case using `pytest.mark.is_staging_test()` if successful.
*   Returns the test case as-is if `pytest` import fails.

### Notable Dependencies and Requirements

*   Relies on `_run_staging` variable
*   Imports the `unittest`, `pytest` libraries (`but only imports pytest in specific condition`)
*   Implies a setup that uses huggingface.co (via model hub) for staging environment


key: chroma:document
string_value:
CODE:

def is_pipeline_test(test_case):
    """
    Decorator marking a test as a pipeline test. If RUN_PIPELINE_TESTS is set to a falsy value, those tests will be
    skipped.
    """
    if not _run_pipeline_tests:
        return unittest.skip(reason="test is pipeline test")(test_case)
    else:
        try:
            import pytest  # We don't need a hard dependency on pytest in the main library
        except ImportError:
            return test_case
        else:


SUMMARY:
**Summary**

This function is a decorator (`is_pipeline_test`) that marks a test case as belonging to a pipeline. If `RUN_PIPELINE_TESTS` is False, the test is skipped. Otherwise, it tries to import and use Pytest's `@pytest.mark.is_pipeline_test()` marker.

**Key Details:**

* Marks test as pipeline test
* Skips test if `RUN_PIPELINE_TESTS` is False
* Imports Pytest library

**Implementation Notes:**

* Uses `unittest.skip` for skipping if condition is false
* Returns decorated test case with Pytest marker if import successful


key: chroma:document
string_value:
CODE:

def is_agent_test(test_case):
    """
    Decorator marking a test as an agent test. If RUN_TOOL_TESTS is set to a falsy value, those tests will be skipped.
    """
    if not _run_agent_tests:
        return unittest.skip(reason="test is an agent test")(test_case)
    else:
        try:
            import pytest  # We don't need a hard dependency on pytest in the main library
        except ImportError:
            return test_case
        else:


SUMMARY:
**Summary**

The `is_agent_test` function is a decorator that marks a test as an agent test and skips it if `RUN_TOOL_TESTS` is falsy. It uses the `unittest.skip` and `pytest.mark.is_agent_test` functions to implement this functionality.

**Key Details**

* Marks a test as an agent test using `pytest.mark.is_agent_test()`
* Skips tests if `RUN_TOOL TESTING` is falsy, using `unittest.skip`
* Requires no hard dependency on pytest, only imports it conditionally

**Notable Dependencies/Interfaces**

* `_run_agent_tests`: a flag indicating whether to run agent tests
* `unittest.skip`: a function from the unittest framework for skipping tests
* `pytest.mark.is_agent_test`: a decorator from the pytest framework for marking tests as agent tests


key: chroma:document
string_value:
CODE:

def slow(test_case):
    """
    Decorator marking a test as slow.

    Slow tests are skipped by default. Set the RUN_SLOW environment variable to a truthy value to run them.

    """


SUMMARY:
**Summary**

Function: `slow`

Purpose:
Marks a test case as marked for running slowly using the `%s` test marker.

Key Implementation Details:

* Skips test by default unless `_run_slow_tests` environment variable is set to truthy value.
* Uses `unittest.skipUnless` decorater with `RUN_SLOW` marker.

Notable Dependencies:

* `unittest`
* `_run_slow_tests` (environment variable, assuming an external setting)


key: chroma:document
string_value:
CODE:

def tooslow(test_case):
    """
    Decorator marking a test as too slow.

    Slow tests are skipped while they're in the process of being fixed. No test should stay tagged as "tooslow" as
    these will not be tested by the CI.

    """


SUMMARY:
**Summary:**

`tooslow` decorator for marking tests as too slow, skipping them in CI until fixed.

**Key Details:**

* Marks a test case using `unittest.skip`
* Reason for skipping: "test is too slow"
* No tests should be tagged with this reason as they are not tested by CI

**Implementation:**

```
def tooslow(test_case):
    return unittest.skip(reason="test is too slow")(test_case)
```


key: chroma:document
string_value:
CODE:

def skip_if_not_implemented(test_func):
    @functools.wraps(test_func)
    def wrapper(*args, **kwargs):
        try:
            return test_func(*args, **kwargs)
        except NotImplementedError as e:
            raise unittest.SkipTest(f"Test skipped due to NotImplementedError: {e}")



SUMMARY:
**Summary: Skipping Test Functions with Not Implemented Error**

The `skip_if_not_implemented` function is a decorator that wraps a test function, skipping its execution if it raises a `NotImplementedError`. The wrapped test function's original behavior is preserved using `functools.wraps`.

**Key Implementation Details:**

* Uses the `@functools.wraps` decorator to preserve original test function metadata.
* Wraps the original test function with a try-except block, catching `NotImplementedError` exceptions.

**Primary Purpose:** Skipping a test function during execution if it raises a `NotImplementedError`.

**Important Method Signature/Interface:**

* `test_func(*args, **kwargs)`: The original test function to be wrapped and potentially skipped.

**External Dependencies/Requirements:**

* `unittest` library (specifically `SkipTest` exception)
* `functools` library (specifically `wraps` decorator)


key: chroma:document
string_value:
CODE:

def apply_skip_if_not_implemented(cls):
    """
    Class decorator to apply @skip_if_not_implemented to all test methods.
    """
    for attr_name in dir(cls):
        if attr_name.startswith("test_"):
            attr = getattr(cls, attr_name)
            if callable(attr):
                setattr(cls, attr_name, skip_if_not_implemented(attr))


SUMMARY:
**Function Summary**

`apply_skip_if_not_implemented`: Decorator function to apply `skip_if_not_implemented` to all test methods in a class.

**Key Implementation Details:**

* Iterates through attributes of the input class (`cls`)
* Filters and calls `skip_if_not_implemented` on callable attributes with names starting with "test_"
* Modifies the original class by setting decorated attributes

**Notable Dependencies/Requirements:**

* `skip_if_not_implemented` decorator function (not shown)
* No external dependencies mentioned


key: chroma:document
string_value:
CODE:

def custom_tokenizers(test_case):
    """
    Decorator marking a test for a custom tokenizer.

    Custom tokenizers require additional dependencies, and are skipped by default. Set the RUN_CUSTOM_TOKENIZERS
    environment variable to a truthy value to run them.
    """


SUMMARY:
**Summary:**

`custom_tokensize_decorator`: A test decorator marking a test for a custom tokenizer. Requires the `RUN_CUSTOM_TOKENIZERS` environment variable to be set to run. Skips tests by default.

**Key Implementation Details:**

* Decorators are used to mark tests for custom tokenizers.
* Tests can only be run if `RUN_CUSTOM_TOKENIZERS` is set to true (`unittest.skipUnless`).

**Notable Dependencies/Requirements:**

* `_run_custom_tokenizers`: an external function or decorator marking the custom tokenizer functionality.


key: chroma:document
string_value:
CODE:

def require_bs4(test_case):
    """
    Decorator marking a test that requires BeautifulSoup4. These tests are skipped when BeautifulSoup4 isn't installed.
    """


SUMMARY:
**Summary**

`require_bs4` is a decorator function marking test cases that require the installation of BeautifulSoup4. It skips tests when BeautifulSoup4 is not available.

**Key Details**

* Marks `test_case` for skipping unless BeautifulSoup4 is installed
* Uses `unittest.skipUnless` with condition checked by `is_bs4_available()`
* Checks if BeautifulSoup4 is installed and requires it for test execution


key: chroma:document
string_value:
CODE:

def require_galore_torch(test_case):
    """
    Decorator marking a test that requires GaLore. These tests are skipped when GaLore isn't installed.
    https://github.com/jiaweizzhao/GaLore
    """


SUMMARY:
**Function Summary: `require_galore_torch`**

This is a decorator function that marks test cases requiring the installation and availability of the GaLore library. It skips tests if GaLore isn't installed, using the `unittest.skipUnless` context manager.

**Key Details:**

- `test_case`: Input test case to be decorated
- `is_galore_torch_available()`: External dependency function to check for GaLore installation availability

This decorator ensures that tests requiring GaLore are skipped if it's not installed, using `unittest.skipUnless`.


key: chroma:document
string_value:
CODE:

def require_lomo(test_case):
    """
    Decorator marking a test that requires LOMO. These tests are skipped when LOMO-optim isn't installed.
    https://github.com/OpenLMLab/LOMO
    """


SUMMARY:
**Function Summary**

`require_lomo` is a decorator that marks test cases as requiring LOMO, skipping them unless LOMO-optim is installed. 

**Key Details:**
- Decorator skips tests when `is_lomo_available()` returns False
- Uses `unittest.skipUnless` to skip testsconditionally


key: chroma:document
string_value:
CODE:

def require_grokadamw(test_case):
    """
    Decorator marking a test that requires GrokAdamW. These tests are skipped when GrokAdamW isn't installed.
    """


SUMMARY:
**Function Summary:**

```python
require_grokadamw(test_case):
  Decors a test case with the requirement to run only when GrokAdamW is installed.
  
  This function utilizes Python's `unittest` module and skips the decorated test case if GrokAdamW is not available.
```

**Key Implementation Details:**

- Utilizes `unittest.skipUnless` decorator to conditionally skip tests based on `is_grokadamw_available()`.
- Requires presence of `GrokAdamW` to be installed for test execution.

**Notable Dependencies/Requirements:**

- Python's `unittest` module
- `is_grokadamw_available()` function (not shown in this snippet)


key: chroma:document
string_value:
CODE:

def require_schedulefree(test_case):
    """
    Decorator marking a test that requires schedulefree. These tests are skipped when schedulefree isn't installed.
    https://github.com/facebookresearch/schedule_free
    """


SUMMARY:
**Summary**

`require_schedulefree` Decorator:

Marks a test case as requiring ScheduleFree (a unit testing library).
This decorator skippers the test when ScheduleFree is not installed.

**Key Details:**

* Returns an `unittest.skipUnless` wrapper around `is_schedulefree_available()`
* Test case is decorated with "test requires schedulefree" error message

**Implementation Details:**

No additional implementation details are provided, as it's a decorator function. 

**Dependencies and Requirements:**

* Unittest library
* schedule_free (available check via `is_schedulefree_available()` method)


key: chroma:document
string_value:
CODE:

def require_cv2(test_case):
    """
    Decorator marking a test that requires OpenCV.

    These tests are skipped when OpenCV isn't installed.

    """


SUMMARY:
**Summary:**

### `require_cv2` Decorator

Marks a test to require OpenCV, skipping if not available. Skips when `is_cv2_available()` returns `False`.

**Key Implementation Details:**

* Uses `unittest.skipUnless` with `is_cv2_available()` check
* Wraps the original test case (`test_case`) in a skip condition

**Context:** OpenCV tests are tagged for skipping on non-OpenCV environments.


key: chroma:document
string_value:
CODE:

def require_levenshtein(test_case):
    """
    Decorator marking a test that requires Levenshtein.

    These tests are skipped when Levenshtein isn't installed.

    """


SUMMARY:
**Summary:**

This function `require_levenshtein` is a decorator that skips test cases marked as requiring Levenshtein unless the Levenshtein algorithm is installed. It uses `unittest.skipUnless` to conditionally skip tests based on the availability of Levenshtein.

**Key Details:**

* **Function Signature:** `def require_levenshtein(test_case):`
* **Purpose:** Marks a test as requiring Levenshtein and skips it if not available.
* **Implementation:** Uses `unittest.skipUnless` to conditionally skip tests.
* **Notable Dependency:** `is_levenshtein_available()` checks for the availability of the Levenshtein algorithm.

**External Dependencies:**

* `unittest`: The unit testing framework used in Python.
* `is_levenshtein_available()`: A function checking for the availability of the Levenshtein algorithm.


key: chroma:document
string_value:
CODE:

def require_nltk(test_case):
    """
    Decorator marking a test that requires NLTK.

    These tests are skipped when NLTK isn't installed.

    """


SUMMARY:
**Function Summary**

`require_nltk`: Decorator marking test cases that require Natural Language Toolkit (NLTK) installation. Skips tests when NLTK is not available.

**Key Details**

* **Purpose**: Ensures integration with NLTK for required tests.
* **Implementation**: Uses `unittest.skipUnless` to skip test case if NLTK is unavailable.
* **Dependency**: `is_nltk_available()` function, which is not shown in this code snippet.


key: chroma:document
string_value:
CODE:

def require_accelerate(test_case, min_version: str = ACCELERATE_MIN_VERSION):
    """
    Decorator marking a test that requires accelerate. These tests are skipped when accelerate isn't installed.
    """
    return unittest.skipUnless(
        is_accelerate_available(min_version), f"test requires accelerate version >= {min_version}"


SUMMARY:
**Summary:**

`require_accelerate` Decorator
================================

This function is a Python decorator that marks tests requiring `accelerate` to automatically skip if the dependency isn't installed. It ensures test cases use a specified minimum version of `accelerate`.

**Key Implementation Details:**
* Checks for availability of `accelerate` using `is_accelerate_available(min_version)`.
* Skips tests unless `accelerate` meets the minimum version requirement (`>= {min_version}`).

**Method Signature:**
- `require_accelerate(test_case, min_version: str = ACCELERATE_MIN_VERSION)`


key: chroma:document
string_value:
CODE:

def require_gguf(test_case, min_version: str = GGUF_MIN_VERSION):
    """
    Decorator marking a test that requires ggguf. These tests are skipped when gguf isn't installed.
    """
    return unittest.skipUnless(is_gguf_available(min_version), f"test requires gguf version >= {min_version}")(
        test_case


SUMMARY:
**Concise Summary:**

`require_gguf` Decorator
==========================

Marks a test as requiring ggguf (>= `GGUF_MIN_VERSION`) and skips it if not installed. If gguf is available, the test is executed.

### Implementation Details:

* Checks if gguf is available using `is_gguf_available(min_version)`
* Skips test using `unittest.skipUnless` if gguf version doesn't meet minimum requirement

### Key Interface/Function:
```python
def is_gguf_available(min_version: str) -> bool
```

### Notable Dependencies:
- `unittest`
- `GGUF_MIN_VERSION` (version of ggguf required for testing)


key: chroma:document
string_value:
CODE:

def require_fsdp(test_case, min_version: str = "1.12.0"):
    """
    Decorator marking a test that requires fsdp. These tests are skipped when fsdp isn't installed.
    """
    return unittest.skipUnless(is_fsdp_available(min_version), f"test requires torch version >= {min_version}")(
        test_case


SUMMARY:
**Function Summary: `require_fsdp`**

**Purpose:** Decorator to mark tests that require fsdp (FSDP) installation and skip if FSDP is not available.

**Primary Purpose:** Ensure fsdp is installed and compatible with the specified version (`min_version`) before running a test case.

**Key Implementation Details:**

* Uses `unittest.skipUnless` to conditionally skip a test case when FSDP is unavailable.
* Invokes `is_fsdp_available(min_version)` function to check for FSDP availability with the specified minimum version.
* Returns a decorated test case with an error message indicating the version requirement.

**Dependencies:**

* `torch`: referenced by `is_fsdp_available(min_version)`

**External Dependencies:** 

* `torch` package
* `unittest` module


key: chroma:document
string_value:
CODE:

def require_g2p_en(test_case):
    """
    Decorator marking a test that requires g2p_en. These tests are skipped when SentencePiece isn't installed.
    """


SUMMARY:
**Function Summary: `require_g2p_en` Decorator**

* **Purpose**: A test case decorator that skips tests requiring `g2p_en` if the dependency is not installed or available.

**Key Details:**
- `is_g2p_en_available()` function is used to check the availability of g2p_en.
- The decorator uses the `unittest.skipUnless` method to skip tests when the dependency is not available.
- The decorated test case is wrapped with a `test_case` object from the unittest framework.

**Notable Dependencies:**
- `g2p_en`: A library that provides a spell checking model.
- `SentencePiece`: A library used to integrate thespell checking functionality.


key: chroma:document
string_value:
CODE:

def require_safetensors(test_case):
    """
    Decorator marking a test that requires safetensors. These tests are skipped when safetensors isn't installed.
    """


SUMMARY:
**Summary:**

This function is a Python decorator named `require_safetensors` that skips a test case (`test_case`) unless the `is_safetensors_available()` function returns True, indicating the presence of the safetensor library.

Key Details:

* Primary purpose: Skip test cases based on availability of safetensor library.
* Implementation: Uses `unittest.skipUnless` with a condition check from `is_safetensors_available()`.
* Notable dependencies: Safetensor library ( referenced by `is_safetensors_available()` ). 

**No external dependencies listed as it's solely focused on its own internal usage.**


key: chroma:document
string_value:
CODE:

def require_rjieba(test_case):
    """
    Decorator marking a test that requires rjieba. These tests are skipped when rjieba isn't installed.
    """


SUMMARY:
**Summary:**

`require_rjieba` Decorator
=============================

* **Functionality:** Skips test cases that require `rjieba` unless it's installed.
* **Design Choice:** Uses `unittest.skipUnless` to conditionally skip tests based on the availability of `rjieba`.
* **Key Implementation:**
	+ Calls `is_rjieba_available()` to determine if `rjieba` is installed.
	+ Applies `unittest.skipUnless` with an error message if `rjieba` isn't available.

**Code Context:**

This decorator enhances unit testing by marking test cases that rely on `rjieba`. When the test case is executed, it checks if `rjieba` is installed and skips the test only if not found.


key: chroma:document
string_value:
CODE:

def require_jieba(test_case):
    """
    Decorator marking a test that requires jieba. These tests are skipped when jieba isn't installed.
    """


SUMMARY:
**Code Summary**

`require_jieba` is a decorator marking test cases that utilize JIEBA (a Chinese word segmentation library) as mandatory. The decorated tests are skipped if JIEBA is not installed.

**Details**

* Marking tests with `@require_jieba(test_case)` indicates they must use JIEBA for execution.
* Tests requiring JIEBA will be skipped when the library is not available (`is_jieba_available()` returns False).
* This decorator ensures that the test suite can run reliably only if JIEBA is installed.


key: chroma:document
string_value:
CODE:

def require_jinja(test_case):
    """
    Decorator marking a test that requires jinja. These tests are skipped when jinja isn't installed.
    """


SUMMARY:
**Summary:**

* **Purpose and Functionality:** Decorator to mark test cases that require Jinja installation, skipping the test if Jinja isn't available.

**Key Implementation Details:**

* Uses `unittest.skipUnless` to skip the test if Jinja isn't available.

**Notable Dependencies/Requirements:**

* [is_jinja_available()](external func) - checks if Jinja is installed.
* `unittest` - testing framework (built-in Python library).
* `unittest.skipUnless` - decorator skipping tests based on a condition.


key: chroma:document
string_value:
CODE:

def require_tf2onnx(test_case):


SUMMARY:
**Summary:**

`require_tf2onnx` test case modifier ensures that tests requiring `tf2onnx` are only executed when `tf2onnx` is available. It uses `unittest.skipUnless` to skip the test if `is_tf2onnx_available()` returns False.

**Key Details:**

* Modifier function: `require_tf2onnx`
* Purpose: Skips tests if `tf2onnx` is not available
* Implementation: Uses `is_tf2onnx_available()` and `unittest.skipUnless`

**Notable Dependencies:**

* `is_tf2onnx_available()`: A test utility function checking for availability of `tf2onnx`
* `unittest`: Unit testing framework

**Interfaces/Classes:**

* `unittest` (unit testing framework)
* `test_case` (unit test case instance)


key: chroma:document
string_value:
CODE:

def require_onnx(test_case):


SUMMARY:
**Summary:**

`require_onnx` is a test case decorator that uses `unittest.skipUnless` to skip a test if ONNX (Open Neural Network Exchange) support is not available, unless testing is explicitly enabled.

**Key Details:**
- `test_case` parameter indicates the decorated test case.
- Uses `is_onnx_available()` function to determine ONNX availability and `"test requires ONNX"` error message for skipping tests.


key: chroma:document
string_value:
CODE:

def require_timm(test_case):
    """
    Decorator marking a test that requires Timm.

    These tests are skipped when Timm isn't installed.

    """


SUMMARY:
**Function Summary: `require_timm`**

* **Purpose**: A decorator that marks test cases that require the Timm library, skipping them unless Timm is installed.

* **Key Implementation Details**:
	+ Uses the `unittest.skipUnless` function to conditionally skip tests based on Timm availability.
	+ Calls `is_timm_available()` to determine whether Timm is installed or not.

* **Dependencies**:
	+ `unittest` module
	+ `is_timm_available()` function (not shown in this code snippet)

* **Notes**: This decorator is designed to protect tests that rely on Timm from running without it, ensuring that the test suite remains stable and predictable.


key: chroma:document
string_value:
CODE:

def require_natten(test_case):
    """
    Decorator marking a test that requires NATTEN.

    These tests are skipped when NATTEN isn't installed.

    """


SUMMARY:
**Summary**

* Function: `require_natten`
* Purpose: A decorator that skips or runs tests depending on the availability of NATTEN.
* Primary purpose: Indicates which tests require NATTEN installation.

**Key implementation details**

* Uses `unittest.skipUnless` with a call to `is_natten_available()`.
* Utilizes the `test_case` object passed as an argument.

**Notable dependencies or requirements**

* None explicitly mentioned, but relies on:
	+ `unittest` module
	+ `is_natten_available()` function (not shown in this snippet)


key: chroma:document
string_value:
CODE:

def require_torch(test_case):
    """
    Decorator marking a test that requires PyTorch.

    These tests are skipped when PyTorch isn't installed.

    """


SUMMARY:
**Summary:**

`require_torch` Decorator:

*   Marks a test as requiring PyTorch.
*   Skips the test if PyTorch is not available; otherwise, runs it.
*   Utilizes `is_torch_available()` to check for PyTorch installation.

Notable Details:

*   Uses `unittest.skipUnless()` with custom error message.
*   Relies on `is_torch_available()` external function or variable (not defined in this snippet).


key: chroma:document
string_value:
CODE:

def require_flash_attn(test_case):
    """
    Decorator marking a test that requires Flash Attention.

    These tests are skipped when Flash Attention isn't installed.

    """


SUMMARY:
**Concise Summary**

`require_flash_attn` is a decorator that marks a test for requirement of Flash Attention. It skips the test if Flash Attention isn't installed, warning about the missing dependency.

**Key Implementation Details:**

* Used `unittest.skipUnless` to skip test if Flash Attention is not available
* Calls `is_flash_attn_2_available()` interface to check Flash Attention availability

**Dependence:**

* Requires Flash Attention feature (`is_flash_attn_2_available()`)


key: chroma:document
string_value:
CODE:

def require_torch_sdpa(test_case):
    """
    Decorator marking a test that requires PyTorch's SDPA.

    These tests are skipped when requirements are not met (torch version).
    """


SUMMARY:
Here is a concise and focused summary of the provided function code chunk:

**Function Summary: `require_torch_sdpa`**

*   Decorator marking a test that requires availability of PyTorch's SDPA (STeady-state Dynamic Programming Algorithm).
*   Skips the test if PyTorch SDPA is not installed.
*   Tests are skipped with an error message indicating requiring PyTorch SDPA.

Key Details:

- **Function Signature:** `require_torch_sdpa(test_case)`
- **Primary Purpose:** Verifies availability of PyTorch's SDPA and enforces its requirement for tests.
- **Implementation:** Utilizes `unittest.skipUnless` to conditionally skip tests based on the availability of PyTorch SDPA, checked by the `is_torch_sdpa_available()` function.


key: chroma:document
string_value:
CODE:

def require_read_token(fn):
    """
    A decorator that loads the HF token for tests that require to load gated models.
    """
    token = os.getenv("HF_HUB_READ_TOKEN")

    @wraps(fn)
    def _inner(*args, **kwargs):
        if token is not None:
            with patch("huggingface_hub.utils._headers.get_token", return_value=token):
                return fn(*args, **kwargs)
        else:  # Allow running locally with the default token env variable
            return fn(*args, **kwargs)



SUMMARY:
**Summary**

The `require_read_token` function is a decorator that loads the HF token from the environment variable "HF_HUB_READ_TOKEN" before running a target function (`fn`). It allows for loading gated models in tests by overriding the default token behavior. 

**Key Implementation Details**

* Uses `patch` to temporarily override the default token retrieval mechanism
* Returns `fn` with modified headers only if a valid token is set in the environment variable

**Dependencies/Requirements**

* `os` (for accessing environment variables)
* `wraps` (from `functools`) or equivalent decorator for preserving original function signature and docstring
* `patch` (from the testing library, e.g., `unittest.mock`)


key: chroma:document
string_value:
CODE:

def require_peft(test_case):
    """
    Decorator marking a test that requires PEFT.

    These tests are skipped when PEFT isn't installed.

    """


SUMMARY:
**Function Summary**

`require_peft`: A decorator function marking a test case that requires the PEFT library. Skips the test when PEFT is not installed.

**Key Details**

* Decorates a `test_case` with a skip condition using `unittest.skipUnless`.
* Conditionally skips tests unless `is_peft_available()` returns `True`, allowing for installation checks.

**Notable Dependencies**

* `unittest`: The standard Python unit testing framework.
* `is_peft_available()`: A function checking if PEFT is installed (implementation not shown).


key: chroma:document
string_value:
CODE:

def require_torchvision(test_case):
    """
    Decorator marking a test that requires Torchvision.

    These tests are skipped when Torchvision isn't installed.

    """


SUMMARY:
**Summary**

This is a Python decorator function `require_torchvision` that marks a test case (`test_case`) as requiring torchvision installation. When torchvision isn't available, the test is skipped using the `unittest.skipUnless` context manager. 

The primary purpose is to ensure tests are only run if torchvision is installed.

**Key Implementation Details:**

* Uses `unittest.skipUnless` with condition `is_torchvision_available()`
* Affects all test cases marked by the decorator

**Notable Dependencies/Requirements:**

* `torchvision`: required for the test case execution
* `is_torchVisionAvailable()` function (exact name not provided)
       


key: chroma:document
string_value:
CODE:

def require_torch_or_tf(test_case):
    """
    Decorator marking a test that requires PyTorch or TensorFlow.

    These tests are skipped when neither PyTorch not TensorFlow is installed.

    """
    return unittest.skipUnless(is_torch_available() or is_tf_available(), "test requires PyTorch or TensorFlow")(
        test_case


SUMMARY:
**Code Summary**

Adds a decorator to mark tests that require either PyTorch or TensorFlow. The decorated tests are skipped when neither library is installed.

**Key Implementation Details**

* Uses `unittest.skipUnless` decorator
* Checks availability of PyTorch and/or TensorFlow using `is_torch_available()` and `is_tf_available()`

**Primary Purpose**
Requires specific libraries for the test execution.

**Notable Dependencies or Requirements**

* `torch` (PyTorch)
* `tf` (TensorFlow)


key: chroma:document
string_value:
CODE:

def require_intel_extension_for_pytorch(test_case):
    """
    Decorator marking a test that requires Intel Extension for PyTorch.

    These tests are skipped when Intel Extension for PyTorch isn't installed or it does not match current PyTorch
    version.

    """
    return unittest.skipUnless(
        is_ipex_available(),
        "test requires Intel Extension for PyTorch to be installed and match current PyTorch version, see"
        " https://github.com/intel/intel-extension-for-pytorch",


SUMMARY:
**Function Summary: `require_intel_extension_for_pytorch`**

This is a decorator function that marks test cases as skipping when Intel Extension for PyTorch is not installed or incompatible with the current PyTorch version.

**Implementation Details:**

* Utilizes `unittest.skipUnless` to skip tests based on the presence and compatibility of Intel Extension.
* Calls `is_ipex_available()` function to verify availability of Intel Extension.
* Passes a custom error message referencing the official GitHub documentation for Intel Extension installation and compatibility.

**Notable Dependencies/Requirements:**

* `unittest` module
* `is_ipex_available()` function (not included in this code snippet)
* Intel Extension for PyTorch (requires installation and matching current PyTorch version)

Note: The exact implementation dependencies are excluded to maintain focus on the primary purpose of the function.


key: chroma:document
string_value:
CODE:

def require_tensorflow_probability(test_case):
    """
    Decorator marking a test that requires TensorFlow probability.

    These tests are skipped when TensorFlow probability isn't installed.

    """
    return unittest.skipUnless(is_tensorflow_probability_available(), "test requires TensorFlow probability")(
        test_case


SUMMARY:
**TF Probability Test Decorator**

* **Purpose and Functionality**: Tests not compatible with TensorFlow Probability will be skipped when `is_tensorflow_probability_available()` returns False.
* **Key Algorithm/Design Choice**: Uses the `unittest.skipUnless` decorator to conditionally skip tests based on availability of TensorFlow Probability.

**Note:** `is.tensorflow_probability_available()` is assumed to be a function that checks if TensorFlow Probability is installed and available.


key: chroma:document
string_value:
CODE:

def require_torchaudio(test_case):
    """
    Decorator marking a test that requires torchaudio. These tests are skipped when torchaudio isn't installed.
    """


SUMMARY:
**Summary:**
The `require_torchaudio` decorator is a test decorator that skips a test case unless PyTorch Audio (torchaudio) is available.

**Key Implementation Details:**

* Decorates a test case using the `unittest.skipUnless` function.
* Ensures the decorator checks if torchaudio is available by calling the `is_torchaudio_available()` function.
* Skips the test case if torchaudio is not available, and provides a specific error message.

**Notable Dependencies/Requirements:**

* PyTorch Audio (torchaudio) module requires.


key: chroma:document
string_value:
CODE:

def require_tf(test_case):
    """
    Decorator marking a test that requires TensorFlow. These tests are skipped when TensorFlow isn't installed.
    """


SUMMARY:
**Summary**

`require_tf` is a decorator that marks a test case as requiring TensorFlow, skipping the test when TensorFlow is not available. It utilizes `unittest.skipUnless` to check for availability and skips tests if TensorFlow installation is absent.

**Key Details:**

* Decorator marking test cases for TensorFlow dependency
* Skips tests with `"test requires TensorFlow"` error message when TensorFlow is not installed

No notable dependencies or requirements besides TensorFlow, which can be verified using the `is_tf_available` function.


key: chroma:document
string_value:
CODE:

def require_flax(test_case):
    """
    Decorator marking a test that requires JAX & Flax. These tests are skipped when one / both are not installed
    """


SUMMARY:
**Code Summary:**

`require_flax` Decorator
==========================

### Purpose and Functionality

Marks test cases requiring JAX and Flax, skipping tests if dependency is not installed.

### Key Implementation Details:

* Uses `unittest.skipUnless` with the result of `is_flax_available()` to skip test case.
* The decorator can be applied using `@require_flax` syntax.

### Notable Dependencies or Requirements

* Requires `jax`, `flax` libraries for functionality.


key: chroma:document
string_value:
CODE:

def require_sentencepiece(test_case):
    """
    Decorator marking a test that requires SentencePiece. These tests are skipped when SentencePiece isn't installed.
    """


SUMMARY:
**Summary:**

`require_sentencepiece` is a unit test decorator marking test cases that require SentencePiece, skipping them if the library isn't installed.

**Key Details:**

* Decorator skips tests if `is_sentencepiece_available()` returns False
* Tests are skipped with explicit error message "test requires SentencePiece"
* Reverts normal test execution using `unittest.skipUnless`


key: chroma:document
string_value:
CODE:

def require_sacremoses(test_case):
    """
    Decorator marking a test that requires Sacremoses. These tests are skipped when Sacremoses isn't installed.
    """


SUMMARY:
**Summary**

`require_sacremoses` is a test case decorator that skips tests requiring Sacremoses when it's not installed. It wraps the `test_case` with `unittest.skipUnless`, which checks if `is_sacremoses_available()` returns True.

**Key Details**

* Uses `unittest.skipUnless`
* Skips tests when `is_sacremoses_available()` returns False
* Requires Sacremosedebugging

**Dependencies**

* `is_sacremoses_available()`: function checking availability of Sacremoses installation.


key: chroma:document
string_value:
CODE:

def require_seqio(test_case):
    """
    Decorator marking a test that requires SentencePiece. These tests are skipped when SentencePiece isn't installed.
    """


SUMMARY:
**Code Summary: `require_seqio` Decorator**

### Main Purpose and Functionality
The `require_seqio` decorator marks test cases that require installation of the SentencePiece library for execution. When Sentinelpiece isn't installed, these tests are skipped.

### Key Implementation Details

* Uses `unittest.skipUnless` to skip tests if SentensePipe is not available.
* Calls the `is_seqio_available()` function to check for Sentinelpipe availability.

### Notable Dependencies or Requirements
Requires installation of `SentencePiece`.

### Interfaces and Inherited Traits
Extends the behavior of `unittest.skipUnless`.


key: chroma:document
string_value:
CODE:

def require_scipy(test_case):
    """
    Decorator marking a test that requires Scipy. These tests are skipped when SentencePiece isn't installed.
    """


SUMMARY:
**Summary:**

This function `require_scipy` is a decorator used to mark test cases that require the SciPy library. When deployed, it skips the test if SciPy is not available.

**Key Implementation Details:**

* Uses `unittest.skipUnless` with a condition based on `is_scipy_available()`
* Throws an error message "test requires Scipy" if SciPy is unavailable

**Notable Dependencies or Requirements:**

* Requires SciPy library (notably, the `is_scipy_available()` function)
* Inherently ties to `unittest` testing framework
* Implements a decorator pattern to dynamically control test execution


key: chroma:document
string_value:
CODE:

def require_tokenizers(test_case):
    """
    Decorator marking a test that requires 🤗 Tokenizers. These tests are skipped when 🤗 Tokenizers isn't installed.
    """


SUMMARY:
**Summary**

This function, `require_tokenizers`, is a decorator that marks test cases requiring Tokenzie library (not 🤗 Tokenizers as mentioned in the code). It skips tests that are not compatible with Tokenzie when it's not installed.

* **Purpose**: Ensure compatibility of tests with Tokenzie library.
* **Key details**:
	+ Uses `unittest.skipUnless` to skip tests when Tokenzie is unavailable.
	+ Requires Tok McKenzie (`tok McKenzie`) library (noted as Tokenzie in the code comment).
* **Dependencies**:
	+ `Tok McKenzie` library.


key: chroma:document
string_value:
CODE:

def require_tensorflow_text(test_case):
    """
    Decorator marking a test that requires tensorflow_text. These tests are skipped when tensroflow_text isn't
    installed.
    """


SUMMARY:
**Summary:**

`require_tensorflow_text` Decorator:
 
* Marks test cases that require TensorFlow Text installation and automatically skip them if `is_tensorflow_text_available()` returns False.

### Implementation Details:

* Utilizes the `unittest.skipUnless` function to conditionally skip tests based on TensorFlow Text availability.
* Relies on the `is_tensorflow_text_available()` helper function to determine whether TensorFlow Text is installed.

### Notable Dependencies/Requirements:

* `tensorflow_text`: The dependency required for this test case (uninstalling or not installing this package causes the test to be skipped).
* `unittest.skipUnless()`
* `is_tensorflow_text_available()`: An internal function checking the availability of TensorFlow Text.


key: chroma:document
string_value:
CODE:

def require_keras_nlp(test_case):
    """
    Decorator marking a test that requires keras_nlp. These tests are skipped when keras_nlp isn't installed.
    """


SUMMARY:
**Summary**

This is a Python decorator function `require_keras_nlp` that marks test cases requiring the installation of Keras NLP library. It skips tests if `is_keras_nlp_available()` returns False.

**Key Points**

* **Purpose**: Skips non-Keras NLP installed tests.
* **Functionality**:
	+ Uses `unittest.skipUnless` to skip test case if Keras NLP is not available.
	+ Calls `is_keras_nlp_available()` function as a condition for skipping the test.
* **Important details**:
	+ Decorator marks test cases requiring Keras NLP library installation.
	+ Tests are skipped if Keras NLP is not installed.


key: chroma:document
string_value:
CODE:

def require_pandas(test_case):
    """
    Decorator marking a test that requires pandas. These tests are skipped when pandas isn't installed.
    """


SUMMARY:
**Function Summary**

`require_pandas` is a decorator function used to mark test cases that require the presence of pandas. It skips tests if pandas is not available.

* **Purpose:** Marks test case as requiring pandas installation.
* **Key Implementation Details:**
	+ Uses `unittest.skipUnless` with condition check from `is_pandas_available()`.
	+ Decorates the `test_case` parameter with a message indicating requirement of pandas.
* **Notable Dependencies/Requirements:** pandas
* **Interfaces Implemented:** `unittest.TestCase`
* **External Dependencies:** None


key: chroma:document
string_value:
CODE:

def require_pytesseract(test_case):
    """
    Decorator marking a test that requires PyTesseract. These tests are skipped when PyTesseract isn't installed.
    """


SUMMARY:
**Summary:**

`require_pytesseract` is a test case decorator, marked by PyTesseract's installation status. It uses `unittest.skipUnless` to conditionally skip test runs when PyTesseract is unavailable.

**Detailed Analysis:**

* **Primary Purpose:** This decorator checks if PyTesseract is installed and skips the decorated test case if not available.
* **Key Implementation Details:**
	+ `is_pytesseract_available()` function retrieves PyTesseract's availability status.
	+ `unittest.skipUnless` skips the test case only when `is_pytesseract_available()` returns False.
* **Notable Dependencies or Requirements:** PyTesseract is required for this decorator to work.

**Searchable Context:**

* Classes and functions mentioned in this summary:
	+ `require_pytesseract`
	+ `is_pytesseract_available`
	+ `unittest` (unittest module)
	+ `PyTesseract`
* External Dependencies:
	+ PyTesseract (library)
* Inherited or Implemented Interfaces:
	- None


key: chroma:document
string_value:
CODE:

def require_pytorch_quantization(test_case):
    """
    Decorator marking a test that requires PyTorch Quantization Toolkit. These tests are skipped when PyTorch
    Quantization Toolkit isn't installed.
    """
    return unittest.skipUnless(is_pytorch_quantization_available(), "test requires PyTorch Quantization Toolkit")(
        test_case


SUMMARY:
**Summary:**

`require_pytorch_quantization` decorator: skips test case unless `is_pytorch_quantization_available()` returns True. Test fails with error message if PyTorch Quantization Toolkit is not installed.

**Key Details:**

* Function name: `require_pytorch_quantization`
* Purpose: Conditionally skip a test case based on the availability of PyTorch Quantization Toolkit.
* Implemented by: Utilizes `unittest.skipUnless` decorator and calls virtual function `is_pytorch_quantization_available()`.
* Dependent on: Availability of PyTorch Quantization Toolkit (checked via `is_pytorch_quantization_available()`).
 

**Dependencies:**

* `unittest`: used for decorating `test_case`
* `is_pytorch_quantization_available()`: checks availability of PyTorch Quantization Toolkit


key: chroma:document
string_value:
CODE:

def require_vision(test_case):
    """
    Decorator marking a test that requires the vision dependencies. These tests are skipped when torchaudio isn't
    installed.
    """


SUMMARY:
**Summary:**

Requires decorator for test cases marked as needing vision dependencies, skipping tests unless torchaudio is available.

* Purpose: Ensures tests are skipped when torchaudio isn't installed and flags vision-related tests.
* Key implementation details:
	+ Uses `unittest.skipUnless` with the result of `is_vision_available()`.
	+ Wraps the test case in a skip instruction if `is_vision_available()` returns False.
* Dependencies:
	+ `unittest`
	+ `torchaudio`
	+ `is_vision_available()` function (not shown)


key: chroma:document
string_value:
CODE:

def require_ftfy(test_case):
    """
    Decorator marking a test that requires ftfy. These tests are skipped when ftfy isn't installed.
    """


SUMMARY:
**Concise Summary:**

`require_ftfy` Decorator:
 
- Purpose: Skips tests unless `ftfy` is installed and available.
  
- Implementation: Uses `unittest.skipUnless` to conditionally skip test cases based on the availability of `is_ftfy_available()`.

 **Key Details:**
 
- Decorator function marking test cases that require `ftfy`.
- Conditionally skips test case using `unittest.skipUnless` with a default message indicating ftfy availability.


key: chroma:document
string_value:
CODE:

def require_spacy(test_case):
    """
    Decorator marking a test that requires SpaCy. These tests are skipped when SpaCy isn't installed.
    """


SUMMARY:
**Summary**

`require_spacy` is a decorator function that marks a test as skipping only if SpaCy is not available. If SpaCy is installed, the test proceeds normally; otherwise, it is automatically skipped.

**Implementation Details**

* Uses `unittest.skipUnless` to conditionally skip tests based on `is_spacy_available()` return value.
* The test case (`test_case`) remains unchanged.
 

**Notable Dependencies/Requirements**

* `SpaCy` library (`is_spacy_available()`)
* `_unittest` module's `skipUnless` function.


key: chroma:document
string_value:
CODE:

def require_torch_multi_gpu(test_case):
    """
    Decorator marking a test that requires a multi-GPU setup (in PyTorch). These tests are skipped on a machine without
    multiple GPUs.

    To run *only* the multi_gpu tests, assuming all test names contain multi_gpu: $ pytest -sv ./tests -k "multi_gpu"
    """
    if not is_torch_available():
        return unittest.skip(reason="test requires PyTorch")(test_case)

    device_count = get_device_count()



SUMMARY:
**Summary**

`require_torch_multi_gpu` is a decorator that marks tests requiring PyTorch multi-GPU setup. It skips tests if:

* PyTorch is not available
* There's only one GPU available.

**Key Details:**

- Skips tests without PyTorch or single GPU.
- Checks for multiple GPUs (`get_device_count()`) to conditionally apply the `unittest.skipUnless` decorator.
- Assumes test names contain "multi_gpu" for multi-GPU tests (using pytest).

**Notable Dependencies/Requirements:**

- `is_torch_available()` function from `torch`
- `get_device_count()` function from unknown origin
- `unittest` module

**Code Organization:**

* Decorator pattern used to mark tests.
* Uses Pythonic syntax and naming conventions.


key: chroma:document
string_value:
CODE:

def require_torch_multi_accelerator(test_case):
    """
    Decorator marking a test that requires a multi-accelerator (in PyTorch). These tests are skipped on a machine
    without multiple accelerators. To run *only* the multi_accelerator tests, assuming all test names contain
    multi_accelerator: $ pytest -sv ./tests -k "multi_accelerator"
    """
    if not is_torch_available():
        return unittest.skip(reason="test requires PyTorch")(test_case)

    return unittest.skipUnless(backend_device_count(torch_device) > 1, "test requires multiple accelerators")(
        test_case


SUMMARY:
**Function Summary: `require_torch_multi_accelerator`**

* **Purpose**: A PyTorch multi-accelerator test requirement decorator to mark tests that necessitate accelerated computing.
* 
* **Implementation Details**:
  * Checks for availability of PyTorch and skips the test if not supported (`is_torch_available()`).
  * Verifies that at least two accelerators are present in the system and skips the test otherwise (`backend_device_count`) unless an argument is provided to use an accelerator.
  
* **Notable Dependencies**:
  + `torch`: The PyTorch library required for this function to work.
  + `unittest`: Provides the necessary functionality for unit testing (used by `unittest.skip` and `unittest.skipUnless`).


key: chroma:document
string_value:
CODE:

def require_torch_non_multi_gpu(test_case):
    """
    Decorator marking a test that requires 0 or 1 GPU setup (in PyTorch).
    """
    if not is_torch_available():
        return unittest.skip(reason="test requires PyTorch")(test_case)

    import torch



SUMMARY:
**Summary**

`require_torch_non_multi_gpu` is a unit test decorator that checks if PyTorch (with 0 or 1 GPU availability) is available and ensures the test case is only executed on devices with this configuration.

**Key Details**

* Requires PyTorch installation and functional GPU availability (either 0 or 1 device).
* Skips tests when PyTorch is not available.
* Skips tests that require more than one device using `torch.cuda.device_count() < 2`.

**Notable Dependencies/Requirements**

* PyTorch (`import torch`)
* `unittest` module


key: chroma:document
string_value:
CODE:

def require_torch_non_multi_accelerator(test_case):
    """
    Decorator marking a test that requires 0 or 1 accelerator setup (in PyTorch).
    """
    if not is_torch_available():
        return unittest.skip(reason="test requires PyTorch")(test_case)



SUMMARY:
**Summary:**

This decorator checks if the PyTorch library is available and verifies that the number of accelerators set up is either 0 or 1. 

* Checks for availability of PyTorch using `is_torch_available()`.
* Skips test if PyTorch is not available.
* Verifies accelerator setup condition using `backend_device_count(torch_device)`. 
    * Skips test if the number of accelerators set up is more than 1.

**Implementation Details:**

* Uses `unittest.skip()` and `unittest.skipUnless()` decorators for skipping tests.
* References PyTorch-specific functions/tensors:
	+ `is_torch_available()`
	+ `torch_device` (assumed to be a tensor-like object)
	+ `backend_device_count()`


key: chroma:document
string_value:
CODE:

def require_torch_up_to_2_gpus(test_case):
    """
    Decorator marking a test that requires 0 or 1 or 2 GPU setup (in PyTorch).
    """
    if not is_torch_available():
        return unittest.skip(reason="test requires PyTorch")(test_case)

    import torch



SUMMARY:
**Summary**

`require_torch_up_to_2_gpus` is a decorator for marking test cases that require PyTorch with 0, 1, or 2 GPU setup. It checks the availability of PyTorch and skips the test if it's not installed.

**Key Details:**

* **Purpose**: Marks test cases requiring specific PyTorch GPU configurations.
* **Implementation**: Checks `torch` library availability and skips tests if no GPUs are available (<= 2).
* **Notable Dependencies**: `torch`, `unittest`, `is_torch_available`.


key: chroma:document
string_value:
CODE:

def require_torch_up_to_2_accelerators(test_case):
    """
    Decorator marking a test that requires 0 or 1 or 2 accelerator setup (in PyTorch).
    """
    if not is_torch_available():
        return unittest.skip(reason="test requires PyTorch")(test_case)

    return unittest.skipUnless(backend_device_count(torch_device) < 3, "test requires 0 or 1 or 2 accelerators")(
        test_case


SUMMARY:
**Function Summary**

`require_torch_up_to_2_accelerators` Decorator:
Mark a test as requiring either 0 or 1 or 2 accelerator setups (PyTorch).

**Key Details:**

* Checks availability of PyTorch using `is_torch_available()`.
* Skips tests with insufficient accelerators (<3) using `unittest.skipUnless`.
* Returns the original test case if PyTorch is not available.

**Notable Dependencies/Requirements:**

* `torch_device`: returns the device associated with PyTorch.
* `backend_device_count`: counts devices in the backend (PyTorch or not).
* `is_torch_available()`: checks availability of PyTorch.


key: chroma:document
string_value:
CODE:

def require_torch_xla(test_case):
    """
    Decorator marking a test that requires TorchXLA (in PyTorch).
    """


SUMMARY:
**Summary**
-----------

Function `require_torch_xla` is a decorator marking test cases that require availability of TorchXLA (PyTorch) with skipping the test if unavailable.

**Key Details**

*   Primary purpose: Decorator to mark tests as requiring TorchXLA
*   **Decorating Pattern:** Uses `unittest.skipUnless`
*   **Function Signatures:**
    *   `is_torch_xla_available()`: Internal function checking for TorchXLA availability
    *   `unittest.skipUnless(is_torch_xla_available(),...)`: Skips test if TorchXLA is not available

**Notable Dependencies/Requirements**

*   Relied on internal `is_torch_xla_available()` function for TorchXLA check
*   Utilizes `unittest` and its skipping mechanism (`unittest.skipUnless`)


key: chroma:document
string_value:
CODE:

def require_torch_neuroncore(test_case):
    """
    Decorator marking a test that requires NeuronCore (in PyTorch).
    """
    return unittest.skipUnless(is_torch_neuroncore_available(check_device=False), "test requires PyTorch NeuronCore")(
        test_case


SUMMARY:
**Summary:**

* **Purpose**: Marks a test case as requiring PyTorch NeuronCore and automatically skips the test if it is not available on the specified device.
* **Notable Details**:
	+ Uses `unittest.skipUnless` decorator to skip tests that do not have PyTorch NeuronCore installed.
	+ Calls `is_torch_neuroncore_available` with `check_device=False` to determine availability without checking device supports.
* **Key Implementation**:
	+ The `is_torch_neuroncore_available` function is called to check if the required library is available.

This summary highlights the main functionality of the code while focusing on its purpose, key implementation details, and notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

def require_torch_npu(test_case):
    """
    Decorator marking a test that requires NPU (in PyTorch).
    """


SUMMARY:
**Summary**

A decorator function `require_torch_npu` marks a test case as requiring PyTorch's NPU functionality. It skips the test unless PyTorch NPU is available.

**Key Details:**

* Function signature: `def require_torch_npu(test_case):`
* Primary purpose: Marks a test case as requiring NPU (PyTorch)
* Notable dependencies:
	+ `is_torch_npu_available()`: function to check NPU availability
	+ `unittest.skipUnless()`: testing decorator to skip tests without NPU


key: chroma:document
string_value:
CODE:

def require_torch_multi_npu(test_case):
    """
    Decorator marking a test that requires a multi-NPU setup (in PyTorch). These tests are skipped on a machine without
    multiple NPUs.

    To run *only* the multi_npu tests, assuming all test names contain multi_npu: $ pytest -sv ./tests -k "multi_npu"
    """
    if not is_torch_npu_available():
        return unittest.skip(reason="test requires PyTorch NPU")(test_case)



SUMMARY:
**Summary**

`require_torch_multi_npu` Decorator:

Mark test cases that require PyTorch's multi-NPU setup.

* Checks if NPU support is available (PyTorch) and skips if not.
* Skips tests with only one NPU or no NPUs assigned.
* Can be used to filter and run tests that depend on multiple NPUs.

**Key Implementation Details**

* Uses `unittest.skip` for skipping tests without NPU support.
* Uses `torch.npu.device_count()` to check number of NPUs available.
* Passes test case with reason when checking if NPU setup meets requirements.


key: chroma:document
string_value:
CODE:

def require_torch_xpu(test_case):
    """
    Decorator marking a test that requires XPU (in PyTorch).

    These tests are skipped when XPU backend is not available. XPU backend might be available either via stock
    PyTorch (>=2.4) or via Intel Extension for PyTorch. In the latter case, if IPEX is installed, its version
    must match match current PyTorch version.
    """


SUMMARY:
**Summary**

This function `require_torch_xpu` is a decorator that marks tests as skippable unless the PyTorch XPU (Accelerated Compute Unit) backend is available. It checks for XPU availability using the `is_torch_xpu_available()` function and conditionally applies the default configuration to the test case if XPU is unavailable.

**Key Implement Details**

* The decorator uses `unittest.skipUnless` to mark tests as skippable.
* It relies on the `is_torch_xpu_available()` function to check for XPU availability.

**Notable Dependencies/Requirements**

* PyTorch.
* `unittest`.
* `is_torch_xpu_available()`.


key: chroma:document
string_value:
CODE:

def require_non_xpu(test_case):
    """
    Decorator marking a test that should be skipped for XPU.
    """


SUMMARY:
**Summary**

`require_non_xpu` is a decorator function that skips a `unittest.testCase` instance if running on an XPU device. It checks the value of `torch_device` and applies the `unittest.skipUnless` marker with a custom failure message.

**Key Details**

* Decorator function applies `unittest.skipUnless` to a test case
* Checks `torchDevice != "xpu"` condition for skipping
* Custom failure message: `"test requires a non-XPU"`

**No notable dependencies or requirements beyond the given code.**


key: chroma:document
string_value:
CODE:

def require_torch_multi_xpu(test_case):
    """
    Decorator marking a test that requires a multi-XPU setup (in PyTorch). These tests are skipped on a machine without
    multiple XPUs.

    To run *only* the multi_xpu tests, assuming all test names contain multi_xpu: $ pytest -sv ./tests -k "multi_xpu"
    """
    if not is_torch_xpu_available():
        return unittest.skip(reason="test requires PyTorch XPU")(test_case)



SUMMARY:
**Summary:**

`require_torch_multi_xpu` Decorator:

* Marks a test that relies on PyTorch's multi-XPU feature
* Skips tests without multiple available XPUs or with invalid device count
* Run all multi-xpu tests using `$ pytest -sv ./tests -k "multi_xpu"`

**Key Implementation Details:**

* Checks availability of PyTorch XPU and ensures proper device count (> 1)
* Uses `unittest.skip` and `skipsUnless` to skip or run tests based on conditions

Notable Dependencies/Requirements:

* `torch.xpu`
* `unittest`
* `pytest`


key: chroma:document
string_value:
CODE:

def require_torchdynamo(test_case):
    """Decorator marking a test that requires TorchDynamo"""


SUMMARY:
**Summary:**

**Function Purpose and Functionality:** `require_torchdynamo` is a decorator for marking test cases that require TorchDynamo.

**Key Implementation Details:**

* Emulates `unittest.skipUnless` to skip tests unless TorchDynamo is available
* Uses `is_torchdynamo_available()` to determine availability

**Notable Dependencies or Requirements:**

* `is_torchdynamo_available()`: a function checking whether TorchDynamo is installed and accessible


key: chroma:document
string_value:
CODE:

def require_torchao(test_case):
    """Decorator marking a test that requires torchao"""


SUMMARY:
**Code Summary**

`require_torchao` is a decorator that marks a test case as requiring `torchao`. It wraps the `unittest.skipUnless` context manager with the condition to check if `torchao` is available. If unavailable, the test will be skipped.

**Key Details:**

* Decorator marking test as requiring `torchao`
* Uses `unittest.skipUnless` to skip test if `torchao` is not available
* Condition checked by the `is_torchao_available()` function


key: chroma:document
string_value:
CODE:

def require_torch_tensorrt_fx(test_case):
    """Decorator marking a test that requires Torch-TensorRT FX"""


SUMMARY:
**Summary:**

`require_torch_tensorrt_fx` Decorator
=====================================


Marks a test as requiring Torch-TensorRT FX functionality. If Torch-TensorRT FX is not available, the test is skipped.

**Key Implementation Details:**

*   Uses `unittest.skipUnless` to ensure the test only runs if Torch-TensorRT FX is enabled.
*   Calls `is_torch_tensorrt_fx_available()` to check for availability of Torch-TensorRT FX functionality.

**Notable Dependencies/Requirements:**

*   `torch`
*   `torch.tortensortfx` ( Torch-TensorRT FX module)
*   `unittest` framework

Note: The actual import is not shown in this snippet, but it's assumed to be part of the larger test framework.


key: chroma:document
string_value:
CODE:

def require_torch_gpu(test_case):
    """Decorator marking a test that requires CUDA and PyTorch."""


SUMMARY:
**Summary:**

`require_torch_gpu` Decorator:

* A decorator function for marking tests that require CUDA and PyTorch support.

**Key Details:**

* **Function Signature:** `require_torch_gpu(test_case)`
* **Purpose:** Checks if the test case is running on a device other than CUDA. If not, skips the test.
* **Imported Dependencies:**
	+ `unittest`
	+ `torch_device` (assumed to be a variable)
	+ `torch` (imported implicitly)
* **Notable Interface:** Uses the `unittest.skipUnless` function with a custom error message.

**Class/Variable Name:** `test_case`

Note that this summary focuses on the key aspects of the code, omitting unnecessary details such as the exact implementation or surrounding context.


key: chroma:document
string_value:
CODE:

def require_torch_gpu_if_bnb_not_multi_backend_enabled(test_case):
    """
    Decorator marking a test that requires a GPU if bitsandbytes multi-backend feature is not enabled.
    """
    if is_bitsandbytes_available() and is_bitsandbytes_multi_backend_available():
        return test_case


SUMMARY:
**Summary:**

`require_torch_gpu_if_bnb_not_multi_backend_enabled` Decorator:

* Requires GPU if `bitsandbytes` multi-backend feature is not enabled.

**Key Implementation Details:**

* Checks availability of `bitsandbytes` and its multi-backend functionality
* Returns the test case unchanged if both are available
* Otherwise, calls `require_torch_gpu` with the test case

**Notable Dependencies/Requirements:**

* `is_bitsandbytes_available()`
* `is_bitsandbytes_multi_backend_available()`
* `require_torch_gpu(test_case)`


key: chroma:document
string_value:
CODE:

def require_torch_accelerator(test_case):
    """Decorator marking a test that requires an accessible accelerator and PyTorch."""
    return unittest.skipUnless(torch_device is not None and torch_device != "cpu", "test requires accelerator")(
        test_case


SUMMARY:
**Summary:**

```
Decorates a unit test to run only if PyTorch is executed on an accelerator device (not CPU) using the `torch_device` variable.
```   

**Key Implementation Details:**

* Utilizes the `unittest.skipUnless` decorator to skip tests without a valid `torch_device`.
* Requires access to both `torch.device` and its value, which must be non-`cpu`.

**Notable Dependencies/Requirements:**

* `torch`: PyTorch library.
* `unittest`: Unit testing framework.
```


key: chroma:document
string_value:
CODE:

def require_torch_fp16(test_case):
    """Decorator marking a test that requires a device that supports fp16"""
    return unittest.skipUnless(
        is_torch_fp16_available_on_device(torch_device), "test requires device with fp16 support"


SUMMARY:
**Summary:**

`require_torch_fp16` Decorator
=====================================

This decorator marks a unit test for execution only if the specified device supports Floating-Point 16 (fp16). It checks if `torch_device` is compatible with fp16 using the `is_torch_fp16_available_on_device` function.

**Implementation Details:**

* Requires torch 1.4+ due to its implementation of fp16 arithmetic.
* Utilizes the `unittest.skipUnless` context manager to execute the test only when the device supports fp16.

**Dependencies and Requirements:**

* torch (version 1.4+) or later, due to support for fp16 precision.


key: chroma:document
string_value:
CODE:

def require_fp8(test_case):
    """Decorator marking a test that requires supports for fp8"""
    return unittest.skipUnless(is_accelerate_available() and is_fp8_available(), "test requires fp8 support")(
        test_case


SUMMARY:
**Summary**

`require_fp8` Decorator:
------------------------

*   Purpose: Marks tests that rely on FP8 (16-bit floating-point) support as skipable if required components are not available.

**Implementation Details**
---------------------------

*   Uses `unittest.skipUnless` to conditionally skip the test case unless two prerequisites are met:
    *   `is_accelerate_available()`
    *   `is_fp8_available()`

These dependencies check for the availability of corresponding features in the system.


key: chroma:document
string_value:
CODE:

def require_torch_bf16(test_case):
    """Decorator marking a test that requires a device that supports bf16"""
    return unittest.skipUnless(
        is_torch_bf16_available_on_device(torch_device), "test requires device with bf16 support"


SUMMARY:
**Summary**

`require_torch_bf16` Decorator: Marks a test as requiring a device that supports Torch BF16. The decorator skips the test unless the required device is available and supports TF32/F16.

**Key Details**

* Test case wrapper using `unittest.skipUnless`
* Checks availability of the required device with `is_torch_bf16_available_on_device(torch_device)`
* Requires Torch BF16 support on device

**External Dependencies**

* `torch_device`: Represents the specific tensor device (e.g., GPU, CPU) being used.
* `is_torch_bf16_available_on_device`: Function checks if the specified device has BF16 support.


key: chroma:document
string_value:
CODE:

def require_torch_bf16_gpu(test_case):
    """Decorator marking a test that requires torch>=1.10, using Ampere GPU or newer arch with cuda>=11.0"""
    return unittest.skipUnless(
        is_torch_bf16_gpu_available(),
        "test requires torch>=1.10, using Ampere GPU or newer arch with cuda>=11.0",


SUMMARY:
**Function Summary:**

`require_torch_bf16_gpu` is a decorator that skips the test case if PyTorch (torch) version 1.10 or higher is not installed, and it is running on an Ampere GPU or a newer architecture with CUDA version 11.0 or higher.

**Key Details:**

* Decorator function
* Requires PyTorch >= 1.10
* Checks for Ampere GPU or newer architecture with CUDA >= 11.0

**Notable Dependencies and Requirements:**

* torch (PyTorch)
	+ Version >= 1.10
* cuda (CUDA)
	+ Version >= 11.0


key: chroma:document
string_value:
CODE:

def require_torch_bf16_cpu(test_case):
    """Decorator marking a test that requires torch>=1.10, using CPU."""
    return unittest.skipUnless(
        is_torch_bf16_cpu_available(),
        "test requires torch>=1.10, using CPU",


SUMMARY:
**Function Summary:**

`require_torch_bf16_cpu` Decorator:
---------------------------

This function is used as a decorator to mark tests that require PyTorch version 1.10 or higher (`torch>=1.10`) and run on CPU.

**Key Details:**

* Checks availability of `bf16` support in PyTorch using `is_torch_bf16_cpu_available()`.
* Skips test if `bf16` is not available, with a warning message indicating the required PyTorch version.

**Notable Dependencies:**

* `unittest`
* `is_torch_bf16_cpu_available()` (referenced class/function not shown in this snippet)

Note: The implementation of `is_torch_bf16_cpu_available()` is not provided in this code chunk.


key: chroma:document
string_value:
CODE:

def require_deterministic_for_xpu(test_case):
    if is_torch_xpu_available():
        return unittest.skipUnless(is_torch_deterministic(), "test requires torch to use deterministic algorithms")(
            test_case
        )
    else:


SUMMARY:
**Summary:**

`require_deterministic_for_xpu` function:
 
- Skips or runs a test case based on availability of PyTorch XPU engine.
- Returns a `unittest.skipUnless` wrapper for tests that require PyTorch deterministic algorithms.

**Key Details:**

- Conditionally checks for PyTorch XPU engine availability using `is_torch_xpu_available()`.
- Skips the test (`return unittest.skipUnless(...)`) if both PyTorch XPU and deterministic algorithms are available.
- Otherwise, runs the test unchanged (`return test_case`).

**Notable Dependencies:**

- `unittest`
- `torch.xpu`
    - `is_torch_xpu_available()`
    - `is_torch_deterministic()`


key: chroma:document
string_value:
CODE:

def require_torch_tf32(test_case):
    """Decorator marking a test that requires Ampere or a newer GPU arch, cuda>=11 and torch>=1.7."""
    return unittest.skipUnless(
        is_torch_tf32_available(), "test requires Ampere or a newer GPU arch, cuda>=11 and torch>=1.7"


SUMMARY:
**Summary**

`require_torch_tf32` Decorator:

* A test case decorator marking the test as skipping if ` Torch-TF32` framework is not available.
* Requires:
	+ GPUs with Ampere or newer arch (torch>=1.7).
	+ CUDA 11 or higher (`cuda>=11`).
* Conditionally skips test execution using `unittest.skipUnless`.

**Implementation Details**

* Invokes the `is_torch_tf32_available()` function to check availability.
* Uses `unittest.skipunless` with a custom condition to determine whether to skip the test.

Note: Exactly preserved class and method names (`test_case`, `unittest`) and references to dependent methods (`is_torch_tf32_available()`).


key: chroma:document
string_value:
CODE:

def require_detectron2(test_case):
    """Decorator marking a test that requires detectron2."""


SUMMARY:
**Summary:**
 
This function, `require_detectron2`, is a decorator used to mark a test as requiring Detectron 2. It skips the test unless `is_detectron2_available()` returns True.

**Key Details:**

- Decorator marking tests that require detectron2
- Skips test if detectron2 is not available

**Notable Dependencies/Requirements:**

- `is_detectron2_available()`
- `unittest`


key: chroma:document
string_value:
CODE:

def require_faiss(test_case):
    """Decorator marking a test that requires faiss."""


SUMMARY:
**Function Summary: `require_faiss`**

This function acts as a decorator to mark a test case that requires the Faiss library to be available.

**Implementation Details:**

- Utilizes `unittest.skipUnless` to skip testing if Faiss is not available and displays an error message.
- Calls the `is_faiss_available()` function before testing, indicating its reliance on the result of this external check.

**Key Dependencies/Requirements:**

- Relies on `faiss` being installed and available for testing.


key: chroma:document
string_value:
CODE:

def require_optuna(test_case):
    """
    Decorator marking a test that requires optuna.

    These tests are skipped when optuna isn't installed.

    """


SUMMARY:
**Summary:**

`require_optuna` Decorator
=============================

**Purpose and Functionality**

Marks a test as requiring Optuna, skipping it unless Optuna is installed.

**Key Implementation Details**

* Skips test using `unittest.skipUnless` if Optuna is not available.
* Uses the `is_optuna_available()` function to check for Optuna installation.

**Notable Dependencies or Requirements**

* `optuna`: dependency required for Optunas availability check
* `unittest`: testing framework used by the decorator.

**Method Signatures and Interfaces**

None explicitly defined, relying on custom `is_optuna_available()` function.


key: chroma:document
string_value:
CODE:

def require_ray(test_case):
    """
    Decorator marking a test that requires Ray/tune.

    These tests are skipped when Ray/tune isn't installed.

    """


SUMMARY:
**Summary**

 decorate function `require_ray`:

*   Marks test cases that require `Ray/tune`.
*   Skips tests when `is_ray_available()` returns False.

**Key Implementation Details**

*   Uses `unittest.skipUnless` decorator to conditionally skip tests based on availability of `Ray/tune`.
*   Calls `is_ray_available()` function to determine the availability of `Ray/tune`.

**Notable Dependencies or Requirements**

*   Utilizes `unittest` module for testing framework support.
*   Relies on `is_ray_available()` function to check if `Ray/tune` is installed.


key: chroma:document
string_value:
CODE:

def require_sigopt(test_case):
    """
    Decorator marking a test that requires SigOpt.

    These tests are skipped when SigOpt isn't installed.

    """


SUMMARY:
**Summary:**

`require_sigopt` is a decorator that marks a test case as skipping until SigOpt availability. It checks if SigOpt is installed and uses the `unittest.skipUnless` function to skip tests unless SigOpt is available.

**Key Details:**

* Primary purpose: Skip test cases when SigOpt isn't installed.
* Implementation:
	+ Checks if SigOpt is available using the `is_sigopt_available()` function.
	+ Uses `unittest.skipUnless` with a custom error message ("test requires SigOpt").
	+ Wraps the decorated test case in this context manager.

**Notable Dependencies:**

* `unittest`: used for skipping tests
* `is_sigopt_available()`: function checking SigOpt availability (not shown in this code snippet)

Note: This summary preserves the function's name, function signature, and main purpose, while extracting key implementation details and dependencies.


key: chroma:document
string_value:
CODE:

def require_wandb(test_case):
    """
    Decorator marking a test that requires wandb.

    These tests are skipped when wandb isn't installed.

    """


SUMMARY:
**Function Summary**

`require_wandb` Decorator
==========================

Mark test case for execution with WandB logging enabled. Skips tests without WandB installation.

### Key Implementation Details

* Uses `unittest.skipUnless` to skip tests without WandB availability
* Invokes `is_wandb_available()` function to check WandB installation status

### Notable Dependencies and Requirements

* `unittest`: Unit testing framework
* `is_wandb_available()`: Function checking WandB installation status


key: chroma:document
string_value:
CODE:

def require_clearml(test_case):
    """
    Decorator marking a test requires clearml.

    These tests are skipped when clearml isn't installed.

    """


SUMMARY:
**Summary**

`require_clearml` Decorator:

* Marks a test case for skipping if ClearML is not installed.
* Uses `unittest.skipUnless` with the result of `is_clearml_available()` to determine whether to skip the test.

**Key Details**

* Requires ClearML installation (`is_clearml_available()`)
* Skips test when ClearML not available
* Utilizes `unittest.skipUnless` decorator


key: chroma:document
string_value:
CODE:

def require_soundfile(test_case):
    """
    Decorator marking a test that requires soundfile

    These tests are skipped when soundfile isn't installed.

    """


SUMMARY:
**Summary: `require_soundfile` Decorator**

### Purpose and Functionality
The `require_soundfile` decorator marks a test case as requiring the `soundfile` library. If `soundfile` is not installed or available, the decorated test case is skipped.

### Key Implementation Details

* Uses `unittest.skipUnless` to skip tests when `soundfile` is unavailable.
* Calls `is_soundfile_available()` to check availability of `soundfile`.
* Returns a decorated test case that skips if `soundfile` is not available.

### Notable Dependencies or Requirements
* `soundfile` library (imported by calling `is_soundfile_available()`)


key: chroma:document
string_value:
CODE:

def require_deepspeed(test_case):
    """
    Decorator marking a test that requires deepspeed
    """


SUMMARY:
**Summary:**

`require_deepspeed` is a test case decorator that ensures a test only runs if DeepSpeed is available. It uses the `unittest.skipUnless` context manager to skip the test if DeepSpeed is not present.

**Key Details:**

* Ensures test suite checks for DeepSpeed availability before execution.
* Uses `is_deepspeed_available()` function to determine availability.
* Deciphers as a wrapper around `skipUnless` to conditionally enable or disable individual test cases.


key: chroma:document
string_value:
CODE:

def require_apex(test_case):
    """
    Decorator marking a test that requires apex
    """


SUMMARY:
**Summary:**

`require_apex`: A decorator function that marks a test case as requiring Apex, skipping the test unless Apex is available.

**Key Details:**

* Utilizes `unittest.skipUnless` to skip the test if Apex is not available.
* Relies on the `is_apex_available()` function to determine Apex availability.

**Notable Dependencies:**

* `unittest`
* `is_apex_available()`: A separate function indicating whether Apex is available.


key: chroma:document
string_value:
CODE:

def require_aqlm(test_case):
    """
    Decorator marking a test that requires aqlm
    """


SUMMARY:
**Summary**

`require_aqlm`: A decorator function marking a test case as requiring `aqlm` (Advanced Query Language Macro) functionality. It skips the test if `aqlm` is not available.

**Key Details:**

* **Function Signature:** `def require_aqlm(test_case):`
* **Primary Purpose:** To mark necessary dependencies
* **Implementation Notes:** Uses the `unittest.skipUnless` decorator to conditionally skip tests based on the `is_aqlm_available()` function.
* **Notable Dependencies:** None specified, but relies on `aqlm`.
* **Related Interfaces:** None.


key: chroma:document
string_value:
CODE:

def require_vptq(test_case):
    """
    Decorator marking a test that requires vptq
    """


SUMMARY:
**Function Summary**

`require_vptq` is a decorator that marks a test case as requiring VPTQ (Version-Patient-Target-Quota) functionality. If VPTQ is not available, the decorated test case will be skipped using `unittest.skipUnless`.

**Key Implementation Details:**

* `is_vptq_available()` function is called to determine whether VPTQ is available
* The `unittest.skipUnless` context manager is used to skip the test case if VPTQ is not available
* Decorated test case is returned

**Notable Dependencies:**

* `unittest` module (built-in Python library)
* `is_vptq_available()` function (unknown implementation, possibly defined elsewhere in the codebase)


key: chroma:document
string_value:
CODE:

def require_eetq(test_case):
    """
    Decorator marking a test that requires eetq
    """
    eetq_available = is_eetq_available()
    if eetq_available:
        try:
            import eetq  # noqa: F401
        except ImportError as exc:
            if "shard_checkpoint" in str(exc):
                # EETQ 1.0.0 is currently broken with the latest transformers because it tries to import the removed
                # shard_checkpoint function, see https://github.com/NetEase-FuXi/EETQ/issues/34.
                # TODO: Remove once eetq releases a fix and this release is used in CI
                eetq_available = False


SUMMARY:
**Summary**

The `require_eetq` function is a decorator that marks test cases requiring EETQ (Elastic Enriched Test Query). It checks availability of `is_eetq_available()` and conditional imports or overrides based on exceptions thrown. The decorated test case is skipped unless EETQ is available, with a corresponding error message.

**Key Implementation Details**

* **Decoration**: Marks a test case to be skipped if EETQ is not available.
* **Dependency Check**: `is_eetq_available()` function checks EETQ availability.
* **Conditional Import**: Attempts to import `eetq` module; overrides availability based on exceptions thrown.

**Notable Dependencies or Requirements**

* `is_eetq_available()`: Checks if EETQ is available.
* `eetq`: Import attempt for EETQ module.


key: chroma:document
string_value:
CODE:

def require_av(test_case):
    """
    Decorator marking a test that requires av
    """


SUMMARY:
**Summary:**

* Purpose: Decorates aunittest test case to skip unless AV is available.
* Functionality: Uses `unittest.skipUnless` and calls `is_av_available()` to determine the required availability of AV.
* Key details:
	+ Calls `is_av_available()`
	+ Skips test if AV is not available
	+ Raises an error message "test requires av" if AV is not available

**Notable dependencies:**
* `unittest`
* `is_av_available()`


key: chroma:document
string_value:
CODE:

def require_bitsandbytes(test_case):
    """
    Decorator marking a test that requires the bitsandbytes library. Will be skipped when the library or its hard dependency torch is not installed.
    """
    if is_bitsandbytes_available() and is_torch_available():
        try:
            import pytest

            return pytest.mark.bitsandbytes(test_case)
        except ImportError:
            return test_case
    else:


SUMMARY:
**Function Summary:**

`require_bitsandbytes`
========================

**Purpose:** Marks a test case as requiring the `bitsandbytes` library. Skips tests when the library is not installed.

**Implementation Details:**

*   Uses a decorator to conditionally apply the `pytest.mark.bitsandbytes` marker.
*   Verifies availability of both `bitsandbytes` and its hard dependency, `torch`, before attempting import.
*   Returns the original test case in two scenarios:
    *   When the required libraries are available.
    *   Otherwise, skips the test with a provided reason.

**Key Components:**

*   `is_bitsandbytes_available()` and `is_torch_available()` checks for library availability.
*   `pytest.mark.bitsandbytes` decorator marker application.
*   `unittest.skip` method usage for skipping tests.

**Example Usage:**
```python
@require_bitsandbytes('test_case')
def my_test():
    # test code here
    pass
```
In this example, the `my_test` function will be skipped if the `bitsandbytes` library is not installed or its hard dependency (`torch`) is missing.


key: chroma:document
string_value:
CODE:

def require_optimum(test_case):
    """
    Decorator for optimum dependency
    """


SUMMARY:
**Summary:**

`require_optimum` is a decorator that skips test cases unless `isOptimumAvailable()` returns True. It uses the `unittest.skipUnless` context manager to conditionally skip tests.

**Key Details:**

* Purpose: Conditionally skip non-optimal test cases
* Implementation:
	+ Calls `isOptimumAvailable()`
	+ Uses `unittest.skipUnless` to skip non-optimal tests if `is_optimum_available()` returns False
	+ Returns decorated `test_case`

**Notable Dependencies/Interfaces:**

* `unittest`
* `is_optimum_available()`: a function within the same module being analyzed


key: chroma:document
string_value:
CODE:

def require_tensorboard(test_case):
    """
    Decorator for `tensorboard` dependency
    """


SUMMARY:
**Summary**

Decorates a test case to skip unless the `tensorboard` dependency is available.

**Key Implementation Details**

* Applies `unittest.skipUnless` with condition checking if `is_tensorboard_available()` returns True
* Throws an error message "test requires tensorboard" otherwise

**Notable Dependencies or Requirements**

* `is_tensorboard_available()` function (interface)
* External dependency on the `tensorboard` library not explicitly managed within this snippet, but suggested by context.


key: chroma:document
string_value:
CODE:

def require_auto_gptq(test_case):
    """
    Decorator for auto_gptq dependency
    """


SUMMARY:
**Summary**

* **Purpose**: Applies the `unittest.skipUnless` decorator to skip test cases unless the `is_auto_gptq_available()` function returns True, indicating the presence of an auto-GPTQ dependency.
* **Key Implementation Details**:
	+ Uses `unittest.skipUnless` with a custom failure message referencing `is_auto_gptq_available()`.
* **Dependencies/Requirements**: `unittest`, `is_auto_gptq_available()` (assuming a separate function definition)


key: chroma:document
string_value:
CODE:

def require_auto_awq(test_case):
    """
    Decorator for auto_awq dependency
    """


SUMMARY:
**Summary**

`require_auto_awq` is a decorator factory that conditionally skips tests using `unittest.skipUnless` if the `auto_awq` dependency is not available.

**Key Implementation Details**

* Conditionally skips tests based on availability of `auto_awq`
* Uses `is_auto_awq_available()` as the criterion
* Invokes `unittest.skipUnless` with the result

**Notable Dependencies or Requirements**

* `unit_test`: implicitly referenced through `unittest`

No external dependencies are listed.


key: chroma:document
string_value:
CODE:

def require_optimum_quanto(test_case):
    """
    Decorator for quanto dependency
    """


SUMMARY:
**Code Summary:**

**Decorator for Checking Quanto Dependency in Unit Test Cases**

* Purpose: Verifies if the "quanto" dependency is available before running unit test cases using the `unittest` framework.
* Primary Functionality: Skips unit tests unless the "is_optimum_quanto_available()" function returns True.

**Implementation Details:**

* Uses a decorator to wrap the original unit test case in a conditional skip.
* Calls the `is_optimum_quanto_available()` function to check for availability of the "quanto" dependency.


key: chroma:document
string_value:
CODE:

def require_compressed_tensors(test_case):
    """
    Decorator for compressed_tensors dependency
    """


SUMMARY:
**Function Summary: `require_compressed_tensors`**

* **Purpose:** A decorator function that skips unit test execution if `compressed_tensors` is not available.

**Key Implementation Details:**
* Uses a decorator pattern with `unittest.skipUnless`.
* Checks availability of `is_compressed_tensors_available()` function as a requirement.
* Passed `test_case` object to be skipped or executed based on the result.

**Dependencies and Requirements:**
* Requires `compressed_tensors` dependency (`is_compressed_tensors_available()` function).
* Inherently depends on `unittest` framework.


key: chroma:document
string_value:
CODE:

def require_fbgemm_gpu(test_case):
    """
    Decorator for fbgemm_gpu dependency
    """


SUMMARY:
**Summary**

`require_fbgemm_gpu` is a decorator function that skips test execution unless the `fbgemm_gpu` dependency is available. It calls `is_fbgemm_gpu_available()` to check for availability and uses the result to conditionally skip the test.

**Key Details:**

* Decorates test cases with skipping behavior
* Checks availability of `fbgemm_gpu`
* Conditionally skips tests based on availability

**Notable Dependencies/Requirements:**

* `unittest` (testing framework)
* `is_fbgemm_gpu_available()` function
* `fbgemm-gpu` dependency


key: chroma:document
string_value:
CODE:

def require_flute_hadamard(test_case):
    """
    Decorator marking a test that requires higgs and hadamard
    """
    return unittest.skipUnless(
        is_flute_available() and is_hadamard_available(), "test requires flute and fast_hadamard_transform"


SUMMARY:
**Summary:**

`require_flute_hadamard` Decorator:

* Marks a test case requiring Higgs and Hadamard availability (using `is_flute_available()` and `is_hadamard_available()` checks).
* Skips the test unless both conditions are met, with an error message indicating required dependencies (`flute` and `fast_hadamard_transform`).

**Key Details:**

* Decorator function for marking test cases.
* Uses `unittest.skipUnless` to conditionally skip tests based on availability checks.
* Calls `is_flute_available()` and `is_hadamard_available()` functions (details not provided in this snippet).


key: chroma:document
string_value:
CODE:

def require_phonemizer(test_case):
    """
    Decorator marking a test that requires phonemizer
    """


SUMMARY:
**Summary:**

`require_phonemizer` is a decorator function that skips a test unless `is_phonemizer_available()` returns `True`, ensuring the phonemizer is available for tests.

**Key Implementation Details:**

* Decorator function `require_phonemizer` uses `unittest.skipUnless()` to skip or run tests based on the availability of `phonemizer`.
* Checks if `is_phonemizer_available()` method returns `True` before executing the test.

**Notable Dependencies or Requirements:**

* `unittest`
* `is_phonemizer_available()`, a related function (not shown in this code snippet)


key: chroma:document
string_value:
CODE:

def require_pyctcdecode(test_case):
    """
    Decorator marking a test that requires pyctcdecode
    """


SUMMARY:
**Function Summary**

`require_pyctcdecode` is a decorator function used to mark tests that require the `pyctcdecode` library. It skips the test if `pyctcdecode` is not available.

**Key Implementation Details**

- **Decorator Function**: Marks a test case (`test_case`) as required for `pyctcdecode` using `unittest.skipUnless`.
- **Availability Check**: Validates `pyctcdecode` availability with `is_pyctcdecode_available()`.

**Notable Dependencies/Requirements**

- `pyctcdecode`: The Python implementation of CTC (Concurrent-Time-Correcting) decoding
- `unittest`: A built-in Python testing framework used for running the test case

**Interfaces Implementations**

- None explicitly mentioned, but implies compatibility with `unittest` framework.


key: chroma:document
string_value:
CODE:

def require_librosa(test_case):
    """
    Decorator marking a test that requires librosa
    """


SUMMARY:
**Summary**

```
[librosa Availability Check Decorator]
 decorate a test with `unittest.skipUnless`,
 ensuring it's skipped if librosa is not available.

Requires `is_librosa_available()` function.
```

**Breakdown**

* Main purpose: To mark tests that require the presence of librosa
* Key implementation detail: Uses `unittest.skipUnless` to conditionally skip a test based on librosa availability
* Notable dependency: `librosa` library, checked using the `is_librosa_available()` function


key: chroma:document
string_value:
CODE:

def require_liger_kernel(test_case):
    """
    Decorator marking a test that requires liger_kernel
    """


SUMMARY:
**Summary:**

Decorates a test case to skip execution unless the `liger_kernel` is available. Utilizes the `unittest.skipUnless` decorator with a custom test condition evaluated by `is_liger_kernel_available()`.

**Key Details:**

* Primary purpose: Decorate test cases that require `liger_kernel`.
* Implementation details:
	+ Uses `unittest.skipUnless` decorator.
	+ Custom test condition is checked using `is_liger_kernel_available()`.


key: chroma:document
string_value:
CODE:

def require_essentia(test_case):
    """
    Decorator marking a test that requires essentia
    """


SUMMARY:
**Summary:**

`require_essentia` is a decorator function marking tests as requiring the Essentia library. It skips the test unless `is_essentia_available()` indicates that Essentia is available, and wraps the test case with `unittest.skipUnless` to achieve this.

**Key Details:**

* Marks tests as requiring Essentia using decorator syntax
* Skips the test if Essentia is not available (via `is_essentia_available()`)
* Uses `unittest skippedUnless` to conditionally skip the test

No notable dependencies or requirements are listed in this code snippet.


key: chroma:document
string_value:
CODE:

def require_pretty_midi(test_case):
    """
    Decorator marking a test that requires pretty_midi
    """


SUMMARY:
**Summary:**

This function is a decorator for unit tests (`unittest`) that checks if the `pretty_midi` library is available. If it's not, the test is skipped using `unittest.skipUnless`.

* **Main Purpose:** Enforces the usage of `pretty_midi` in individual test cases.
* **Key Implementation Details:**
	+ Uses `@unittest.skipUnless` decorator to skip tests without `pretty_midi`.
	+ Calls helper function `is_pretty_midi_available()` to check library availability.
* **Notable Dependencies/Requirements:** `pretty_midi`, `unittest`.


key: chroma:document
string_value:
CODE:

def cmd_exists(cmd):


SUMMARY:
**Summary:**

### Function Purpose

The `cmd_exist` (corrected to match the provided function name) function checks if a specific command (`cmd`) exists in the system.

### Primary Purpose

Uses `shutil.which` to verify the presence of an executable file or command in the current environment, returning `True` if found and `False` otherwise.

### Key Implementation Details:

- Utilizes `shutil.which(cmd)`: searches for the specified command.
- Returns a boolean value (`is not None`) indicating existence.

### Notable Dependencies/Requirements:

- Requires `shutil`, a built-in Python module for interacting with the file system.

### Interface and Interfaces:

None specified, but inherits from standard Python functions and types.


key: chroma:document
string_value:
CODE:

def require_usr_bin_time(test_case):
    """
    Decorator marking a test that requires `/usr/bin/time`
    """


SUMMARY:
**Function Summary**

`require_usr_bin_time`: A decorator function used to mark tests that require running with `/usr/bin/time`. It checks if the command exists and skips the test otherwise.

**Key Implementation Details**

* Uses `unittest.skipUnless` to conditionally skip tests based on command existence.
* References external dependency: `cmd_exists("/usr/bin/time")`.

**Notable Dependencies**

* No explicit dependencies, but assumes existence of `cmd_exists()` function.


key: chroma:document
string_value:
CODE:

def require_sudachi(test_case):
    """
    Decorator marking a test that requires sudachi
    """


SUMMARY:
**Summary:**

Function `require_sudachi` is a decorator that skips a test unless the `is_sudachi_available()` function returns True.

**Key Details:**

* Checks availability of `sudachi` functionality using `is_sudachi_available()`
* Skips test if sudachi is not available, otherwise runs with the provided test case (`test_case`)
* Returns a decorated test case that can be used in unit testing (using `unittest.skipUnless` method)


key: chroma:document
string_value:
CODE:

def require_sudachi_projection(test_case):
    """
    Decorator marking a test that requires sudachi_projection
    """
    return unittest.skipUnless(is_sudachi_projection_available(), "test requires sudachi which supports projection")(
        test_case


SUMMARY:
**Code Summary:**

`require_sudachi_projection` Decorator

**Purpose:** Marks test cases that require sudachi_projection library, skipping tests if not available.

**Functionality:**

* Takes a unit test case as input `test_case`
* Returns a decorated test case using `unittest.skipUnless`, conditional on availability of `is_sudachi_projection_available()`
* Raises an exception if `sudachi_projection` is not supported

**Key Implementation Details:**

* Uses `unittest.skipUnless` to conditionally skip tests
* Calls `is_sudachi_projection_available()` method, which is assumed to be a separate function defining the library availability check.

**Notable Dependencies and Requirements:**

* `unittest`
* `sudachi_projection` library (checked for availability using an external method `is_sudachi_projection_available()`)


key: chroma:document
string_value:
CODE:

def require_jumanpp(test_case):
    """
    Decorator marking a test that requires jumanpp
    """


SUMMARY:
**Function Summary**

`require_jumanpp` is a test decorator that skips a test unless `jumanpp` is available. It uses the `unittest` framework and relies on the existence of a function `is_jumanpp_available()` to determine availability.

**Key Implementation Details**

* Skips a test if `jumanpp` is not available
* Uses the `unittest.skipUnless` context manager to achieve this

**Notable Dependencies/Requirements**

* `unittest`
* `is_jumanpp_available() function` (exact implementation not shown)


key: chroma:document
string_value:
CODE:

def require_cython(test_case):
    """
    Decorator marking a test that requires jumanpp
    """


SUMMARY:
**Summary:**

This function, `require_cython`, is a decorator used to mark tests that require the Cython compiler. It skips the test if Cython is not available and displays an error message.

**Key Implementation Details:**

- Uses the `unittest.skipUnless` context manager to skip the test if Cython is not available.
- Calls the `is_cython_available()` function, which is not shown in this snippet.

**Notable Dependencies/Requirements:**

- Utilizes the `unittest` and `is_cython_available()` modules (not shown).
- Expects the availability of Cython compiler.


key: chroma:document
string_value:
CODE:

def require_tiktoken(test_case):
    """
    Decorator marking a test that requires TikToken. These tests are skipped when TikToken isn't installed.
    """


SUMMARY:
**Function Summary**

`require_tiktoken` Decorator
==========================

Marks a test case as requiring TikToken. Tests with this decorator are skipped if TikToken is not installed.

Functions:
* `is_tiktoken_available()`: checks if TikToken is installed (not shown in this code snippet)
* `unittest.skipUnless()`: skips the test if `is_tiktoken_available()` returns False

Key Implementation Details:

* Uses a decorator to mark tests that require TikToken
* Skips tests if TikToken is not available due to `unittest.skipUnless`


key: chroma:document
string_value:
CODE:

def get_gpu_count():
    """
    Return the number of available gpus (regardless of whether torch, tf or jax is used)
    """
    if is_torch_available():
        import torch

        return torch.cuda.device_count()
    elif is_tf_available():
        import tensorflow as tf

        return len(tf.config.list_physical_devices("GPU"))
    elif is_flax_available():
        import jax

        return jax.device_count()
    else:


SUMMARY:
**Function Summary: `get_gpu_count`**

Retrieves the number of available GPUs, leveraging various frameworks (torch, tf, jax) or defaulting to 0 if none is supported.

### Key Functionality:

- Returns GPU count using:
  - `torch.cuda.device_count()` when torch is available
  - `len(tf.config.list_physical_devices("GPU"))` when tf is available
  - `jax.device_count()` when flax/jax is available

### Notable Dependencies/Requirements:

- `torch`
- `tensorflow` (tf)
- `jax` (or `flax`)
- `is_torch_available()`, `is_tf_available()`, and `is_flax_available()` internal utility functions (not shown in this code snippet)


key: chroma:document
string_value:
CODE:

def get_tests_dir(append_path=None):
    """
    Args:
        append_path: optional path to append to the tests dir path

    Return:
        The full path to the `tests` dir, so that the tests can be invoked from anywhere. Optionally `append_path` is
        joined after the `tests` dir the former is provided.

    """
    # this function caller's __file__
    caller__file__ = inspect.stack()[1][1]
    tests_dir = os.path.abspath(os.path.dirname(caller__file__))

    while not tests_dir.endswith("tests"):
        tests_dir = os.path.dirname(tests_dir)

    if append_path:
        return os.path.join(tests_dir, append_path)
    else:


SUMMARY:
**Function Summary**

`get_tests_dir(append_path=None)`:

Returns the full path to the `tests` directory. optionally appends `append_path` to it.

**Key Implementation Details:**

* Uses caller's file path (`caller__file__`) to determine the tests dir
* Ensures the resulting path ends with "tests" using a loop

**Notable Dependencies/Requirements:**

* `inspect`
* `os`

**Primary Purpose:** Returns an absolute path to the `tests` directory.


key: docstring
string_value:
#
# Helper functions for dealing with testing text outputs
# The original code came from:
# https://github.com/fastai/fastai/blob/master/tests/utils/text.py
# When any function contains print() calls that get overwritten, like progress bars,
# a special care needs to be applied, since under pytest -s captured output (capsys
# or contextlib.redirect_stdout) contains any temporary printed strings, followed by
# \r's. This helper function ensures that the buffer will contain the same output
# with and without -s in pytest, by turning:
# foo bar\r tar mar\r final message
# into:
# final message
# it can handle a single string or a multiline buffer


key: chroma:document
string_value:
CODE:
# it can handle a single string or a multiline buffer
def apply_print_resets(buf):


SUMMARY:
**Function Summary: `apply_print_resets(buf)`**

**Purpose and Functionality:**
Removes trailing `\r` characters from the end of a given input buffer (`buf`) to ensure consistent behavior with and without `pytest -s` captor output.

**Implementation Details:**

* Uses `re.sub()` to replace trailing `\r` characters with an empty string.
* Sets `re.M` flag for multiline matching.
* Returns the modified buffer (`buf`).

**Notable Dependencies/Requirements:**
- Utilizes `re` (regular expression) module.
- Relies on `pytest` framework with `-s` option.

**Interfaces/Methods:**
- `re.sub()`: replaces substrings in a given string.

Note: The code uses a helper function from an external repository, but the implementation details are self-contained within this summary.


key: chroma:document
string_value:
CODE:

def assert_screenout(out, what):
    out_pr = apply_print_resets(out).lower()
    match_str = out_pr.find(what.lower())


SUMMARY:
**Summary**

Function `assert_screenout(out, what)` asserts that a specified output string (`out`) contains the expected strings (`what`). It normalizes the output and searches for matching values.

**Key Details**

- Normalizes output using `apply_print_resets(out)` before searching.
- Searches for lower-case version of input (`what`).
- Asserts presence of match string.


key: chroma:document
string_value:
CODE:

def set_model_tester_for_less_flaky_test(test_case):
    target_num_hidden_layers = 1
    # TODO (if possible): Avoid exceptional cases
    exceptional_classes = [
        "ZambaModelTester",
        "RwkvModelTester",
        "AriaVisionText2TextModelTester",
        "GPTNeoModelTester",
        "DPTModelTester",
    ]
    if test_case.model_tester.__class__.__name__ in exceptional_classes:
        target_num_hidden_layers = None
    if hasattr(test_case.model_tester, "out_features") or hasattr(test_case.model_tester, "out_indices"):
        target_num_hidden_layers = None

    if hasattr(test_case.model_tester, "num_hidden_layers") and target_num_hidden_layers is not None:
        test_case.model_tester.num_hidden_layers = target_num_hidden_layers
    if (
        hasattr(test_case.model_tester, "vision_config")
        and "num_hidden_layers" in test_case.model_tester.vision_config
        and target_num_hidden_layers is not None
    ):
        test_case.model_tester.vision_config = copy.deepcopy(test_case.model_tester.vision_config)
        test_case.model_tester.vision_config["num_hidden_layers"] = target_num_hidden_layers
    if (
        hasattr(test_case.model_tester, "text_config")
        and "num_hidden_layers" in test_case.model_tester.text_config
        and target_num_hidden_layers is not None
    ):
        test_case.model_tester.text_config = copy.deepcopy(test_case.model_tester.text_config)
        test_case.model_tester.text_config["num_hidden_layers"] = target_num_hidden_layers

    # A few model class specific handling

    # For Albert
    if hasattr(test_case.model_tester, "num_hidden_groups"):


SUMMARY:
**Summary: `set_model_tester_for_less_flaky_test` Function**

* **Purpose:** Adjusts model tester's configuration to reduce flakiness in tests for specific models (e.g., ZambaModelTester, AriaVisionText2TextModelTester).
* **Functionality:**
	+ Toggles or sets `num_hidden_layers` based on presence of certain attributes.
	+ Copies and overwrites `vision_config` and `text_config` dictionaries as needed.

**Key Implementation Details:**

* Iterates through exceptional classes (e.g., `ZambaModelTester`) and sets `target_num_hidden_layers` to `None` if the condition is met.
* Modifies `num_hidden_layers`, `num_hidden_groups`, and attributes of related configurations (`vision_config` and `text_config`) based on presence of certain attributes.

**Notable Dependencies/Requirements:**

* `copy`
* Model-specific configurations (e.g., `ZambaModelTester`) with attributes:
	+ `num_hidden_layers`
	+ `num_hidden_groups`
	+ `vision_config`
	+ `text_config`


key: chroma:document
string_value:
CODE:

def set_config_for_less_flaky_test(config):
    target_attrs = [
        "rms_norm_eps",
        "layer_norm_eps",
        "norm_eps",
        "norm_epsilon",
        "layer_norm_epsilon",
        "batch_norm_eps",
    ]
    for target_attr in target_attrs:
        setattr(config, target_attr, 1.0)

    # norm layers (layer/group norm, etc.) could cause flaky tests when the tensors have very small variance.
    # (We don't need the original epsilon values to check eager/sdpa matches)
    attrs = ["text_config", "vision_config", "text_encoder", "audio_encoder", "decoder"]
    for attr in attrs:
        if hasattr(config, attr):
            for target_attr in target_attrs:


SUMMARY:
**Function Summary: `set_config_for_less_flaky_test`**

**Purpose:** This function sets default values for specific configuration attributes (`rms_norm_eps`, `layer_norm_eps`, etc.) in the provided `config` object.

**Key Implementation Details:**

*   Sets default norm epsilon values for all applicable configuration attributes.
*   Iterates through attribute lists and recursively updates nested configurations.

**External Dependencies/Notable Requirements:**

*   `config` object must be a dictionary-like structure.
*   The `setattr` function is used to dynamically update the configuration attributes.

This function appears to be part of a test configuration adjustment mechanism, aiming to reduce flaky test issues caused by negligible tensor variances in norm layers.


key: chroma:document
string_value:
CODE:

def set_model_for_less_flaky_test(model):
    # Another way to make sure norm layers have desired epsilon. (Some models don't set it from its config.)
    target_names = ("LayerNorm", "GroupNorm", "BatchNorm", "RMSNorm", "BatchNorm2d", "BatchNorm1d")
    target_attrs = ["eps", "epsilon", "variance_epsilon"]
    if is_torch_available() and isinstance(model, torch.nn.Module):
        for module in model.modules():
            if type(module).__name__.endswith(target_names):
                for attr in target_attrs:
                    if hasattr(module, attr):


SUMMARY:
**Summary:**

`set_model_for_less_flaky_test(model)` 

* Sets the desired norms with `epsilon` values to various models, ensuring norm layers have predictable behavior.

**Key Implementation Details:**

- Loops through model's modules and sets `eps`, `epsilon`, or `variance_epsilon` attributes of norm layers to 1.0.
  
**Notable Dependencies or Requirements:**

- `torch.nn.Module` and `torch` library for availability check (`is_torch_available()`)


key: chroma:document
string_value:
CODE:

class CaptureStd:
    """
    Context manager to capture:

        - stdout: replay it, clean it up and make it available via `obj.out`
        - stderr: replay it and make it available via `obj.err`

    Args:
        out (`bool`, *optional*, defaults to `True`): Whether to capture stdout or not.
        err (`bool`, *optional*, defaults to `True`): Whether to capture stderr or not.
        replay (`bool`, *optional*, defaults to `True`): Whether to replay or not.
            By default each captured stream gets replayed back on context's exit, so that one can see what the test was
            doing. If this is a not wanted behavior and the captured data shouldn't be replayed, pass `replay=False` to
            disable this feature.

    Examples:

    ```python
    # to capture stdout only with auto-replay
    with CaptureStdout() as cs:
        print("Secret message")
    assert "message" in cs.out

    # to capture stderr only with auto-replay
    import sys

    with CaptureStderr() as cs:
        print("Warning: ", file=sys.stderr)
    assert "Warning" in cs.err

    # to capture both streams with auto-replay
    with CaptureStd() as cs:
        print("Secret message")
        print("Warning: ", file=sys.stderr)
    assert "message" in cs.out
    assert "Warning" in cs.err

    # to capture just one of the streams, and not the other, with auto-replay
    with CaptureStd(err=False) as cs:
        print("Secret message")
    assert "message" in cs.out
    # but best use the stream-specific subclasses

    # to capture without auto-replay
    with CaptureStd(replay=False) as cs:
        print("Secret message")
    assert "message" in cs.out
    ```"""

    def __init__(self, out=True, err=True, replay=True):
        self.replay = replay

        if out:
            self.out_buf = StringIO()
            self.out = "error: CaptureStd context is unfinished yet, called too early"
        else:
            self.out_buf = None
            self.out = "not capturing stdout"

        if err:
            self.err_buf = StringIO()
            self.err = "error: CaptureStd context is unfinished yet, called too early"
        else:
            self.err_buf = None
            self.err = "not capturing stderr"

    def __enter__(self):
        if self.out_buf:
            self.out_old = sys.stdout
            sys.stdout = self.out_buf

        if self.err_buf:
            self.err_old = sys.stderr
            sys.stderr = self.err_buf

        return self

    def __exit__(self, *exc):
        if self.out_buf:
            sys.stdout = self.out_old
            captured = self.out_buf.getvalue()
            if self.replay:
                sys.stdout.write(captured)
            self.out = apply_print_resets(captured)

        if self.err_buf:
            sys.stderr = self.err_old
            captured = self.err_buf.getvalue()
            if self.replay:
                sys.stderr.write(captured)
            self.err = captured

    def __repr__(self):
        msg = ""
        if self.out_buf:
            msg += f"stdout: {self.out}\n"
        if self.err_buf:
            msg += f"stderr: {self.err}\n"


SUMMARY:
**Class CaptureStd Context Manager**
=====================================

### Purpose and Functionality

The `CaptureStd` class is a context manager that captures and replays output to `stdout` and `stderr`. It provides flexible management of captured streams, including replaying them back on exit.

### Key Implementation Details

*   Initializes contexts for capturing `stdout` and/or `stderr`, using `StringIO` buffers.
*   Sets up `sys.stdout` and `sys.stderr` to capture output in `__enter__`.
*   Restores original `sys.stdout` and `sys.stderr` values on exit (`__exit__`).
*   Replays captured streams back to the same stream or writes them directly to it, depending on the `replay` flag.
*   Manages stored output via method `apply_print_resets`.

### Notable Dependencies and Requirements

*   Uses `StringIO`, a built-in Python module for easily creating file-like objects as strings.

This code provides a concise way to manage capturing and replaying output streams, simplifying the process of testing and debugging print statements.


key: docstring
string_value:
# in tests it's the best to capture only the stream that's wanted, otherwise
# it's easy to miss things, so unless you need to capture both streams, use the
# subclasses below (less typing). Or alternatively, configure `CaptureStd` to
# disable the stream you don't need to test.


key: chroma:document
string_value:
CODE:

class CaptureStdout(CaptureStd):
    """Same as CaptureStd but captures only stdout"""

    def __init__(self, replay=True):


SUMMARY:
## Class Summary: CaptureStdOut

### Purpose and Functionality

The `CaptureStdOut` class is a subclass of `CaptureStd` that captures only the standard output (`stdout`) stream, inheriting its functionality while filtering out error output. It allows for capturing only the desired stream in tests.

#### Primary Purpose
Captures only `stdout` stream with optional replay feature.

### Key Implementation Details

- Utilizes `super()` to initialize `CaptureStd` with `err=False` and `replay=replay`.
- Includes a docstring advice on not capturing both streams or configuring `CaptureStd` to disable unnecessary streams.

#### Important Method Signatures and Interfaces
- `__init__(self, replay: bool)`: Initializes the object with optional replay feature.
  - `replay: bool`
  
### Dependencies and Requirements

- Subclasses `CaptureStd`
- Requires `super()` for inheritance and method call chaining.


key: chroma:document
string_value:
CODE:

class CaptureStderr(CaptureStd):
    """Same as CaptureStd but captures only stderr"""

    def __init__(self, replay=True):


SUMMARY:
**Summary:**

The `CaptureStderr` class extends the base `CaptureStd` class to specifically capture standard error output. It inherits the functionality of its parent class while modifying the initialization with only stderr enabled.

**Key Details:**

* Extends `CaptureStd` and overrides it with `out=False`
* Initializes with replay enabled (`replay=True`) by default

**Dependencies:**

* Inheriting from `CaptureStd`
* External dependency on `super()` for inheritance


key: chroma:document
string_value:
CODE:

class CaptureLogger:
    """
    Context manager to capture `logging` streams

    Args:
        logger: 'logging` logger object

    Returns:
        The captured output is available via `self.out`

    Example:

    ```python
    >>> from transformers import logging
    >>> from transformers.testing_utils import CaptureLogger

    >>> msg = "Testing 1, 2, 3"
    >>> logging.set_verbosity_info()
    >>> logger = logging.get_logger("transformers.models.bart.tokenization_bart")
    >>> with CaptureLogger(logger) as cl:
    ...     logger.info(msg)
    >>> assert cl.out, msg + "\n"
    ```
    """

    def __init__(self, logger):
        self.logger = logger
        self.io = StringIO()
        self.sh = logging.StreamHandler(self.io)
        self.out = ""

    def __enter__(self):
        self.logger.addHandler(self.sh)
        return self

    def __exit__(self, *exc):
        self.logger.removeHandler(self.sh)
        self.out = self.io.getvalue()

    def __repr__(self):


SUMMARY:
**Summary:`

The `CaptureLogger` class is a context manager that captures the output of a given `logging` logger. It ensures that log messages are redirected to a string stream (`StringIO`) instead of being displayed in the console.

**Key Implementation Details:**

* Redirects log messages to a string stream (`self.io`) using `logging.StreamHandler`.
* Allows capture of log output via attribute `self.out`.
* Implements context manager protocol for use with `with` statement.

**Notable Dependencies/Requirements:**

* `logging` module
* `StringIO` class from `io` module

**Primary Purpose:** Capture and redirect logging messages to a string stream.


key: chroma:document
string_value:
CODE:

class TemporaryHubRepo:
    """Create a temporary Hub repository and return its `RepoUrl` object. This is similar to
    `tempfile.TemporaryDirectory` and can be used as a context manager. For example:

        with TemporaryHubRepo(token=self._token) as temp_repo:
            ...

    Upon exiting the context, the repository and everything contained in it are removed.

    Example:

    ```python
    with TemporaryHubRepo(token=self._token) as temp_repo:
        model.push_to_hub(tmp_repo.repo_id, token=self._token)
    ```
    """

    def __init__(self, namespace: Optional[str] = None, token: Optional[str] = None) -> None:
        self.token = token
        with tempfile.TemporaryDirectory() as tmp_dir:
            repo_id = Path(tmp_dir).name
            if namespace is not None:
                repo_id = f"{namespace}/{repo_id}"
            self.repo_url = huggingface_hub.create_repo(repo_id, token=self.token)

    def __enter__(self):
        return self.repo_url

    def __exit__(self, exc, value, tb):


SUMMARY:
**Summary: Temporary Hub Repository Context Manager**

```python
TemporaryHubRepo:
  - **Purpose:** Create a temporary Hub repository with optional namespace and return its RepoUrl object as a context manager.
 
  Attributes:
    - token (str, Optional): API token for accessing the Hub repository.

  Methods:
    - `__enter__`: Returns the RepoUrl object when entering the context.
    - `__exit__`: Deletes the created Hub repository when exiting the context.

  Key Operations:
   - Creates a temporary directory using `tempfile.TemporaryDirectory`.
   - Generates a unique repository ID from the directory name.
   - Creates and initializes a new Hub repository with the given namespace (if provided) and token.
   - Uses `huggingface_hub.create_repo` to create the repository.
   - Deletes the repository using `delete_repo` when exiting the context.

  Examples:
    ```python
    # Create and exit the context
    with TemporaryHubRepo(token='my_token') as repo_url:
      pass
    
    # With namespace
    with TemporaryHubRepo(namespace='my_namespace', token='my_token') as repo_url:
      pass
    ```

  External Dependencies:
   - `tempfile`
   - `huggingface_hub` (via `create_repo` and `delete_repo`)
```

This summary aims to provide a concise overview of the key aspects of the `TemporaryHubRepo` class, including its purpose, attributes, methods, and external dependencies.


key: chroma:document
string_value:
CODE:

class TestCasePlus(unittest.TestCase):
    """
    This class extends *unittest.TestCase* with additional features.

    Feature 1: A set of fully resolved important file and dir path accessors.

    In tests often we need to know where things are relative to the current test file, and it's not trivial since the
    test could be invoked from more than one directory or could reside in sub-directories with different depths. This
    class solves this problem by sorting out all the basic paths and provides easy accessors to them:

    - `pathlib` objects (all fully resolved):

       - `test_file_path` - the current test file path (=`__file__`)
       - `test_file_dir` - the directory containing the current test file
       - `tests_dir` - the directory of the `tests` test suite
       - `examples_dir` - the directory of the `examples` test suite
       - `repo_root_dir` - the directory of the repository
       - `src_dir` - the directory of `src` (i.e. where the `transformers` sub-dir resides)

    - stringified paths---same as above but these return paths as strings, rather than `pathlib` objects:

       - `test_file_path_str`
       - `test_file_dir_str`
       - `tests_dir_str`
       - `examples_dir_str`
       - `repo_root_dir_str`
       - `src_dir_str`

    Feature 2: Flexible auto-removable temporary dirs which are guaranteed to get removed at the end of test.

    1. Create a unique temporary dir:

    ```python
    def test_whatever(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
    ```

    `tmp_dir` will contain the path to the created temporary dir. It will be automatically removed at the end of the
    test.


    2. Create a temporary dir of my choice, ensure it's empty before the test starts and don't
    empty it after the test.

    ```python
    def test_whatever(self):
        tmp_dir = self.get_auto_remove_tmp_dir("./xxx")
    ```

    This is useful for debug when you want to monitor a specific directory and want to make sure the previous tests
    didn't leave any data in there.

    3. You can override the first two options by directly overriding the `before` and `after` args, leading to the
        following behavior:

    `before=True`: the temporary dir will always be cleared at the beginning of the test.

    `before=False`: if the temporary dir already existed, any existing files will remain there.

    `after=True`: the temporary dir will always be deleted at the end of the test.

    `after=False`: the temporary dir will always be left intact at the end of the test.

    Note 1: In order to run the equivalent of `rm -r` safely, only subdirs of the project repository checkout are
    allowed if an explicit `tmp_dir` is used, so that by mistake no `/tmp` or similar important part of the filesystem
    will get nuked. i.e. please always pass paths that start with `./`

    Note 2: Each test can register multiple temporary dirs and they all will get auto-removed, unless requested
    otherwise.

    Feature 3: Get a copy of the `os.environ` object that sets up `PYTHONPATH` specific to the current test suite. This
    is useful for invoking external programs from the test suite - e.g. distributed training.


    ```python
    def test_whatever(self):
        env = self.get_env()
    ```"""

    def setUp(self):
        # get_auto_remove_tmp_dir feature:
        self.teardown_tmp_dirs = []

        # figure out the resolved paths for repo_root, tests, examples, etc.
        self._test_file_path = inspect.getfile(self.__class__)
        path = Path(self._test_file_path).resolve()
        self._test_file_dir = path.parents[0]
        for up in [1, 2, 3]:
            tmp_dir = path.parents[up]
            if (tmp_dir / "src").is_dir() and (tmp_dir / "tests").is_dir():
                break
        if tmp_dir:
            self._repo_root_dir = tmp_dir
        else:
            raise ValueError(f"can't figure out the root of the repo from {self._test_file_path}")
        self._tests_dir = self._repo_root_dir / "tests"
        self._examples_dir = self._repo_root_dir / "examples"
        self._src_dir = self._repo_root_dir / "src"

    @property
    def test_file_path(self):
        return self._test_file_path

    @property
    def test_file_path_str(self):
        return str(self._test_file_path)

    @property
    def test_file_dir(self):
        return self._test_file_dir

    @property
    def test_file_dir_str(self):
        return str(self._test_file_dir)

    @property
    def tests_dir(self):
        return self._tests_dir

    @property
    def tests_dir_str(self):
        return str(self._tests_dir)

    @property
    def examples_dir(self):
        return self._examples_dir

    @property
    def examples_dir_str(self):
        return str(self._examples_dir)

    @property
    def repo_root_dir(self):
        return self._repo_root_dir

    @property
    def repo_root_dir_str(self):
        return str(self._repo_root_dir)

    @property
    def src_dir(self):
        return self._src_dir

    @property
    def src_dir_str(self):
        return str(self._src_dir)

    def get_env(self):
        """
        Return a copy of the `os.environ` object that sets up `PYTHONPATH` correctly, depending on the test suite it's
        invoked from. This is useful for invoking external programs from the test suite - e.g. distributed training.

        It always inserts `./src` first, then `./tests` or `./examples` depending on the test suite type and finally
        the preset `PYTHONPATH` if any (all full resolved paths).

        """
        env = os.environ.copy()
        paths = [self.src_dir_str]
        if "/examples" in self.test_file_dir_str:
            paths.append(self.examples_dir_str)
        else:
            paths.append(self.tests_dir_str)
        paths.append(env.get("PYTHONPATH", ""))

        env["PYTHONPATH"] = ":".join(paths)
        return env

    def get_auto_remove_tmp_dir(self, tmp_dir=None, before=None, after=None):
        """
        Args:
            tmp_dir (`string`, *optional*):
                if `None`:

                   - a unique temporary path will be created
                   - sets `before=True` if `before` is `None`
                   - sets `after=True` if `after` is `None`
                else:

                   - `tmp_dir` will be created
                   - sets `before=True` if `before` is `None`
                   - sets `after=False` if `after` is `None`
            before (`bool`, *optional*):
                If `True` and the `tmp_dir` already exists, make sure to empty it right away if `False` and the
                `tmp_dir` already exists, any existing files will remain there.
            after (`bool`, *optional*):
                If `True`, delete the `tmp_dir` at the end of the test if `False`, leave the `tmp_dir` and its contents
                intact at the end of the test.

        Returns:
            tmp_dir(`string`): either the same value as passed via *tmp_dir* or the path to the auto-selected tmp dir
        """
        if tmp_dir is not None:
            # defining the most likely desired behavior for when a custom path is provided.
            # this most likely indicates the debug mode where we want an easily locatable dir that:
            # 1. gets cleared out before the test (if it already exists)
            # 2. is left intact after the test
            if before is None:
                before = True
            if after is None:
                after = False

            # using provided path
            path = Path(tmp_dir).resolve()

            # to avoid nuking parts of the filesystem, only relative paths are allowed
            if not tmp_dir.startswith("./"):
                raise ValueError(
                    f"`tmp_dir` can only be a relative path, i.e. `./some/path`, but received `{tmp_dir}`"
                )

            # ensure the dir is empty to start with
            if before is True and path.exists():
                shutil.rmtree(tmp_dir, ignore_errors=True)

            path.mkdir(parents=True, exist_ok=True)

        else:
            # defining the most likely desired behavior for when a unique tmp path is auto generated
            # (not a debug mode), here we require a unique tmp dir that:
            # 1. is empty before the test (it will be empty in this situation anyway)
            # 2. gets fully removed after the test
            if before is None:
                before = True
            if after is None:
                after = True

            # using unique tmp dir (always empty, regardless of `before`)
            tmp_dir = tempfile.mkdtemp()

        if after is True:
            # register for deletion
            self.teardown_tmp_dirs.append(tmp_dir)

        return tmp_dir

    def python_one_liner_max_rss(self, one_liner_str):
        """
        Runs the passed python one liner (just the code) and returns how much max cpu memory was used to run the
        program.

        Args:
            one_liner_str (`string`):
                a python one liner code that gets passed to `python -c`

        Returns:
            max cpu memory bytes used to run the program. This value is likely to vary slightly from run to run.

        Requirements:
            this helper needs `/usr/bin/time` to be installed (`apt install time`)

        Example:

        ```
        one_liner_str = 'from transformers import AutoModel; AutoModel.from_pretrained("google-t5/t5-large")'
        max_rss = self.python_one_liner_max_rss(one_liner_str)
        ```
        """

        if not cmd_exists("/usr/bin/time"):
            raise ValueError("/usr/bin/time is required, install with `apt install time`")

        cmd = shlex.split(f"/usr/bin/time -f %M python -c '{one_liner_str}'")
        with CaptureStd() as cs:
            execute_subprocess_async(cmd, env=self.get_env())
        # returned data is in KB so convert to bytes
        max_rss = int(cs.err.split("\n")[-2].replace("stderr: ", "")) * 1024
        return max_rss

    def tearDown(self):
        # get_auto_remove_tmp_dir feature: remove registered temp dirs
        for path in self.teardown_tmp_dirs:
            shutil.rmtree(path, ignore_errors=True)
        self.teardown_tmp_dirs = []
        if is_accelerate_available():
            AcceleratorState._reset_state()
            PartialState._reset_state()

            # delete all the env variables having `ACCELERATE` in them
            for k in list(os.environ.keys()):
                if "ACCELERATE" in k:


SUMMARY:
This code is a test class written in Python, which provides several utility methods for running tests and measuring CPU memory usage. Here's a concise summary:

**Main Features:**

1. **Temporary Directory Management**: The `get_auto_remove_tmp_dir` method creates a temporary directory with optional custom behavior (e.g., empty before test or kept after test).
2. **CPU Memory Measurement**: The `python_one_liner_max_rss` method runs a Python one-liner code and returns the maximum CPU memory usage in bytes.
3. **Environment Management**: The `get_env` method returns an environment dictionary with relevant settings for testing, including `/usr/bin/time` location.

**Dependent Services:**

1. `/usr/bin/time` is required for measuring CPU memory usage (install it if not available).
2. `Accelerate` framework is used for distributed computing (reset state after test).

Overall, this class provides a set of convenient methods for managing temporary directories and measuring CPU memory usage, making it easier to write tests with these features.


key: chroma:document
string_value:
CODE:

def mockenv(**kwargs):
    """
    this is a convenience wrapper, that allows this ::

    @mockenv(RUN_SLOW=True, USE_TF=False) def test_something():
        run_slow = os.getenv("RUN_SLOW", False) use_tf = os.getenv("USE_TF", False)

    """


SUMMARY:
**Summary**

The `mockenv` function is a convenience wrapper that enables environment variable mocking for testing purposes. It replaces the target environment variables with values provided through keyword arguments (`kwargs`). The function leverages the `unittest.mock.patch.dict` function to achieve this.

**Key Implementation Details:**

* Replaces `os.environ` with mocked value using `patch.dict`
* Allows mocking of key environment variables using `__kwargs__`

**Notable Dependencies/Requirements:**

* `os` module (specifically `os.getenv` and `os.environ`)
* `unittest.mock` module (specifically `patch.dict`)


key: chroma:document
string_value:
CODE:

def pytest_addoption_shared(parser):
    """
    This function is to be called from `conftest.py` via `pytest_addoption` wrapper that has to be defined there.

    It allows loading both `conftest.py` files at once without causing a failure due to adding the same `pytest`
    option.

    """
    option = "--make-reports"
    if option not in pytest_opt_registered:
        parser.addoption(
            option,
            action="store",
            default=False,
            help="generate report files. The value of this option is used as a prefix to report names",
        )


SUMMARY:
**Summary: `pytest_addoption_shared` Function**

This function, called from `conftest.py`, adds a custom option `--make-reports` to the pytest parser. It registers the option with a default value of `False`.

**Key details:**

* **Option signature:** `--make-reports`
* **Action:** Store the option value as an integer (`store`)
* **Default value:** `False`
* **Help message:** Generates report files and uses the option value as a prefix for report names.

**Dependencies/erequisites:**

None external dependencies mentioned. The code assumes the existence of:
- `pytest_opt_registered` (a dictionary to track registered options)
- A `conftest.py` file referencing this function via `pytest_addoption`


key: chroma:document
string_value:
CODE:

def pytest_terminal_summary_main(tr, id):
    """
    Generate multiple reports at the end of test suite run - each report goes into a dedicated file in the current
    directory. The report files are prefixed with the test suite name.

    This function emulates --duration and -rA pytest arguments.

    This function is to be called from `conftest.py` via `pytest_terminal_summary` wrapper that has to be defined
    there.

    Args:
    - tr: `terminalreporter` passed from `conftest.py`
    - id: unique id like `tests` or `examples` that will be incorporated into the final reports filenames - this is
      needed as some jobs have multiple runs of pytest, so we can't have them overwrite each other.

    NB: this functions taps into a private _pytest API and while unlikely, it could break should pytest do internal
    changes - also it calls default internal methods of terminalreporter which can be hijacked by various `pytest-`
    plugins and interfere.

    """
    from _pytest.config import create_terminal_writer

    if not len(id):
        id = "tests"

    config = tr.config
    orig_writer = config.get_terminal_writer()
    orig_tbstyle = config.option.tbstyle
    orig_reportchars = tr.reportchars

    dir = f"reports/{id}"
    Path(dir).mkdir(parents=True, exist_ok=True)
    report_files = {
        k: f"{dir}/{k}.txt"
        for k in [
            "durations",
            "errors",
            "failures_long",
            "failures_short",
            "failures_line",
            "passes",
            "stats",
            "summary_short",
            "warnings",
        ]
    }

    # custom durations report
    # note: there is no need to call pytest --durations=XX to get this separate report
    # adapted from https://github.com/pytest-dev/pytest/blob/897f151e/src/_pytest/runner.py#L66
    dlist = []
    for replist in tr.stats.values():
        for rep in replist:
            if hasattr(rep, "duration"):
                dlist.append(rep)
    if dlist:
        dlist.sort(key=lambda x: x.duration, reverse=True)
        with open(report_files["durations"], "w") as f:
            durations_min = 0.05  # sec
            f.write("slowest durations\n")
            for i, rep in enumerate(dlist):
                if rep.duration < durations_min:
                    f.write(f"{len(dlist)-i} durations < {durations_min} secs were omitted")
                    break
                f.write(f"{rep.duration:02.2f}s {rep.when:<8} {rep.nodeid}\n")

    def summary_failures_short(tr):
        # expecting that the reports were --tb=long (default) so we chop them off here to the last frame
        reports = tr.getreports("failed")
        if not reports:
            return
        tr.write_sep("=", "FAILURES SHORT STACK")
        for rep in reports:
            msg = tr._getfailureheadline(rep)
            tr.write_sep("_", msg, red=True, bold=True)
            # chop off the optional leading extra frames, leaving only the last one
            longrepr = re.sub(r".*_ _ _ (_ ){10,}_ _ ", "", rep.longreprtext, 0, re.M | re.S)
            tr._tw.line(longrepr)
            # note: not printing out any rep.sections to keep the report short

    # use ready-made report funcs, we are just hijacking the filehandle to log to a dedicated file each
    # adapted from https://github.com/pytest-dev/pytest/blob/897f151e/src/_pytest/terminal.py#L814
    # note: some pytest plugins may interfere by hijacking the default `terminalreporter` (e.g.
    # pytest-instafail does that)

    # report failures with line/short/long styles
    config.option.tbstyle = "auto"  # full tb
    with open(report_files["failures_long"], "w") as f:
        tr._tw = create_terminal_writer(config, f)
        tr.summary_failures()

    # config.option.tbstyle = "short" # short tb
    with open(report_files["failures_short"], "w") as f:
        tr._tw = create_terminal_writer(config, f)
        summary_failures_short(tr)

    config.option.tbstyle = "line"  # one line per error
    with open(report_files["failures_line"], "w") as f:
        tr._tw = create_terminal_writer(config, f)
        tr.summary_failures()

    with open(report_files["errors"], "w") as f:
        tr._tw = create_terminal_writer(config, f)
        tr.summary_errors()

    with open(report_files["warnings"], "w") as f:
        tr._tw = create_terminal_writer(config, f)
        tr.summary_warnings()  # normal warnings
        tr.summary_warnings()  # final warnings

    tr.reportchars = "wPpsxXEf"  # emulate -rA (used in summary_passes() and short_test_summary())

    # Skip the `passes` report, as it starts to take more than 5 minutes, and sometimes it timeouts on CircleCI if it
    # takes > 10 minutes (as this part doesn't generate any output on the terminal).
    # (also, it seems there is no useful information in this report, and we rarely need to read it)
    # with open(report_files["passes"], "w") as f:
    #     tr._tw = create_terminal_writer(config, f)
    #     tr.summary_passes()

    with open(report_files["summary_short"], "w") as f:
        tr._tw = create_terminal_writer(config, f)
        tr.short_test_summary()

    with open(report_files["stats"], "w") as f:
        tr._tw = create_terminal_writer(config, f)
        tr.summary_stats()

    # restore:
    tr._tw = orig_writer
    tr.reportchars = orig_reportchars


SUMMARY:
**Function Summary: `pytest_terminal_summary_main`**

Generate multiple reports at the end of test suite run.

**Purpose:** Emulate `-rA pytest` arguments by generating report files with specific formats and content.

**Key Implementation Details:**

* Generates reports for:
	+ Durations
	+ Failures (long, short, line)
	+ Errors
	+ Warnings
* Uses `terminalreporter` to write reports to files in a designated directory.
* Hijacks internal methods of `terminalreporter` to generate custom report layouts.

**Notable Dependencies and Requirements:**

* `_pytest.config`: used to access configuration settings and terminal writer functionality.
* `terminalreporter`: required for writing reports to files.
* `config.option.tbstyle`: controls the format of test results output (auto, short, line).


key: chroma:document
string_value:
CODE:

class _RunOutput:
    def __init__(self, returncode, stdout, stderr):
        self.returncode = returncode
        self.stdout = stdout


SUMMARY:
**Summary**

Class `_RunOutput` encapsulates the output of a runtime process. It stores three attributes:

* `returncode`: The exit status of the process (int)
* `stdout`: The standard output of the process (str or bytes)
* `stderr`: The standard error of the process (str or bytes)

The class facilitates the structured storage and retrieval of this information.

**Implementation Details**

Key implementation details include:

* An initialization method `__init__` that accepts a return code, stdout, and stderr as parameters.
* A docstring comment (`# noqa`) indicating adherence to Python's PEP-257 style guide for docstrings.


key: chroma:document
string_value:
CODE:

async def _read_stream(stream, callback):
    while True:
        line = await stream.readline()
        if line:
            callback(line)
        else:


SUMMARY:
**Summary:**

`_read_stream` Function:

- Purpose: Reads and asynchronously processes a stream of lines using the provided `callback` function.

**Key Implementation Details:**

* Iterates indefinitely until an empty line (`''`) is encountered.
* Calls `callback` with each non-empty read line.

**Dependencies:**

* None explicitly listed; however, assumes access to `stream.readline()` method.

Note: This summary captures the essential functionality and implementation details of the `_read_stream` function.


key: chroma:document
string_value:
CODE:

async def _stream_subprocess(cmd, env=None, stdin=None, timeout=None, quiet=False, echo=False) -> _RunOutput:
    if echo:
        print("\nRunning: ", " ".join(cmd))

    p = await asyncio.create_subprocess_exec(
        cmd[0],
        *cmd[1:],
        stdin=stdin,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        env=env,
    )

    # note: there is a warning for a possible deadlock when using `wait` with huge amounts of data in the pipe
    # https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.asyncio.subprocess.Process.wait
    #
    # If it starts hanging, will need to switch to the following code. The problem is that no data
    # will be seen until it's done and if it hangs for example there will be no debug info.
    # out, err = await p.communicate()
    # return _RunOutput(p.returncode, out, err)

    out = []
    err = []

    def tee(line, sink, pipe, label=""):
        line = line.decode("utf-8").rstrip()
        sink.append(line)
        if not quiet:
            print(label, line, file=pipe)

    # XXX: the timeout doesn't seem to make any difference here
    await asyncio.wait(
        [
            _read_stream(p.stdout, lambda l: tee(l, out, sys.stdout, label="stdout:")),
            _read_stream(p.stderr, lambda l: tee(l, err, sys.stderr, label="stderr:")),
        ],
        timeout=timeout,
    )


SUMMARY:
**Function Summary: `_stream_subprocess`**

**Purpose and Functionality**
-------------------------

The `_stream_subprocess` function asynchronously executes a subprocess using `asyncio.create_subprocess_exec`. It takes optional arguments for input/output streams and timeout.

**Key Implementation Details**
-----------------------------

*   The function uses `tee` to forward output lines to both the main stream and a separate file (stdout/stderr).
*   It waits for the subprocess to finish with a timeout, preventing potential deadlocks.
*   If the subprocess hangs, it falls back to waiting for completion without debug info.

**Notable Dependencies or Requirements**
-----------------------------------------

*   `asyncio`
*   `sys` for printing output
*   `_ReadStream` abstraction (not shown in this code snippet)

The function returns an instance of `_RunOutput`, containing the subprocess' returncode, stdout, and stderr.


key: chroma:document
string_value:
CODE:

def execute_subprocess_async(cmd, env=None, stdin=None, timeout=180, quiet=False, echo=True) -> _RunOutput:
    loop = asyncio.get_event_loop()
    result = loop.run_until_complete(
        _stream_subprocess(cmd, env=env, stdin=stdin, timeout=timeout, quiet=quiet, echo=echo)
    )

    cmd_str = " ".join(cmd)
    if result.returncode > 0:
        stderr = "\n".join(result.stderr)
        raise RuntimeError(
            f"'{cmd_str}' failed with returncode {result.returncode}\n\n"
            f"The combined stderr from workers follows:\n{stderr}"
        )

    # check that the subprocess actually did run and produced some output, should the test rely on
    # the remote side to do the testing
    if not result.stdout and not result.stderr:
        raise RuntimeError(f"'{cmd_str}' produced no output.")



SUMMARY:
**Summary of `execute_subprocess_async` Function:**

**Purpose and Functionality:** 
The `_execute_subprocess_async` function executes a subprocess asynchronously, handling the outcome in an event loop. It captures the standard input, output, and error streams from the subprocess.

**Key Implementation Details:**

*   Uses asynchronous execution with `asyncio.get_event_loop()` and `loop.run_until_complete()`.
*   Calls `_stream_subprocess` internally to manage the subprocess.
*   Verifies the return code and combined error messages from the subprocess.

**Notable Dependencies or Requirements:**

*   *_RunOutput:* A custom class (not shown in this snippet) that encapsulates the output of the subprocess.
*   *_stream_subprocess:* Another function (not shown in this snippet) handling the subprocess execution, which is imported.

This summary aims to preserve key elements of the original function without including external functions and unnecessary details.


key: chroma:document
string_value:
CODE:

def pytest_xdist_worker_id():
    """
    Returns an int value of worker's numerical id under `pytest-xdist`'s concurrent workers `pytest -n N` regime, or 0
    if `-n 1` or `pytest-xdist` isn't being used.
    """
    worker = os.environ.get("PYTEST_XDIST_WORKER", "gw0")
    worker = re.sub(r"^gw", "", worker, 0, re.M)


SUMMARY:
**Summary:**

The `pytest_xdist_worker_id` function returns the numerical ID of a worker under pytest-xdist's concurrent testing framework. It retrieves the worker ID from the environment variable "PYTEST_XDIST_WORKER", cleans it by removing "gw" prefix, and returns it as an integer. If no valid worker ID is found or -n 1 flag is used, it returns 0.

**Key Implementation Details:**

* Uses `os.environ.get()` to retrieve the "PYTEST_XDIST_WORKER" environment variable
* Applies regular expression substitution (`re.sub`) to remove "gw" prefix from the worker ID

**Notable Dependencies and Requirements:**

* `os` and `re` modules are used for environment variable retrieval and regular expression operations, respectively.


key: chroma:document
string_value:
CODE:

def get_torch_dist_unique_port():
    """
    Returns a port number that can be fed to `torch.distributed.launch`'s `--master_port` argument.

    Under `pytest-xdist` it adds a delta number based on a worker id so that concurrent tests don't try to use the same
    port at once.
    """
    port = 29500
    uniq_delta = pytest_xdist_worker_id()


SUMMARY:
**Summary**

`get_torch_dist_unique_port()` returns a unique port number for PyTorch distributed training, combining a base port (29500) with a delta value generated by `pytest_xdist_worker_id()` to prevent concurrent tests from using the same port.

**Key details**

* Port base: 29500
* Uses `pytest_xdist_worker_id()` to generate a unique delta value

**Implementation notes**

* Adds delta value to base port to create unique port number
* Returns port number suitable for `torch.distributed.launch`'s `--master_port` argument


key: chroma:document
string_value:
CODE:

def nested_simplify(obj, decimals=3):
    """
    Simplifies an object by rounding float numbers, and downcasting tensors/numpy arrays to get simple equality test
    within tests.
    """
    import numpy as np

    if isinstance(obj, list):
        return [nested_simplify(item, decimals) for item in obj]
    if isinstance(obj, tuple):
        return tuple([nested_simplify(item, decimals) for item in obj])
    elif isinstance(obj, np.ndarray):
        return nested_simplify(obj.tolist())
    elif isinstance(obj, Mapping):
        return {nested_simplify(k, decimals): nested_simplify(v, decimals) for k, v in obj.items()}
    elif isinstance(obj, (str, int, np.int64)):
        return obj
    elif obj is None:
        return obj
    elif is_torch_available() and isinstance(obj, torch.Tensor):
        return nested_simplify(obj.tolist(), decimals)
    elif is_tf_available() and tf.is_tensor(obj):
        return nested_simplify(obj.numpy().tolist())
    elif isinstance(obj, float):
        return round(obj, decimals)
    elif isinstance(obj, (np.int32, np.float32, np.float16)):
        return nested_simplify(obj.item(), decimals)
    else:


SUMMARY:
**Nested Simplify Function Summary**

### Purpose and Functionality

The `nested_simplify` function simplifies an object by rounding float numbers, downcasting tensors/numpy arrays for equality tests in tests. It recursively processes various data types (e.g., lists, tuples, dictionaries, torch/tf tensors, numpy arrays, float, int).

### Key Implementation Details

* Rounds float numbers using `round()`.
* Downcasts tensors to numpy arrays for equality tests.
* Uses function nesting and recursion.

### Notable Dependencies and Requirements

* `numpy` for numpy array support
* Torch (`torch`) for torch tensor support (if available)
* TensorFlow (`tf`) for tf tensor support (if available)

### Method Signatures and Interfaces

* Applies to various data types, including `list`, `tuple`, `np.ndarray`, `Mapping`, `str`, `int`, `float`
* Utilizes recursive function calls
* Returns simplified object


