# embedding_metadata

key: chroma:document
string_value:
CODE:

def check_onnxruntime_requirements(minimum_version: Version):
    """
    Check onnxruntime is installed and if the installed version match is recent enough

    Raises:
        ImportError: If onnxruntime is not installed or too old version is found
    """
    try:
        import onnxruntime

        # Parse the version of the installed onnxruntime
        ort_version = parse(onnxruntime.__version__)

        # We require 1.4.0 minimum
        if ort_version < ORT_QUANTIZE_MINIMUM_VERSION:
            raise ImportError(
                f"We found an older version of onnxruntime ({onnxruntime.__version__}) "
                f"but we require onnxruntime to be >= {minimum_version} to enable all the conversions options.\n"
                "Please update onnxruntime by running `pip install --upgrade onnxruntime`"
            )

    except ImportError:
        raise ImportError(
            "onnxruntime doesn't seem to be currently installed. "
            "Please install the onnxruntime by running `pip install onnxruntime`"
            " and relaunch the conversion."


SUMMARY:
**Summary**

This function `check_onnxruntime_requirements` checks if the OnnxRuntime library is installed and its version is recent enough to enable advanced conversions. It:

- **Import check**: Imports onnxruntime and attempts to parse its version using the `parse` function (not defined in this snippet).
- **Version comparison**: Compares the parsed version with the minimum required version `ORT_QUANTIZE_MINIMUM_VERSION`.
- **Error handling**:
  - Raises an `ImportError` if OnnxRuntime is not installed.
  - If installed, but with too old a version, raises an `ImportError` urging update of OnnxRuntime via pip.

**Key implementation details:**

* Uses `onnxruntime.__version__`
* Requires minimal version `ORT_QUANTIZE_MINIMUM_VERSION`

**Notable dependencies:**

* External dependency: `onnxruntime` (imported)
* Inherited/implemented interface: None

Note: The `parse` function is not defined in this snippet, implying it's possibly another utility or library not immediately relevant to the described functionality.


key: chroma:document
string_value:
CODE:

def export_pytorch(
    preprocessor: Union["PreTrainedTokenizer", "FeatureExtractionMixin", "ProcessorMixin"],
    model: "PreTrainedModel",
    config: OnnxConfig,
    opset: int,
    output: Path,
    tokenizer: "PreTrainedTokenizer" = None,
    device: str = "cpu",
) -> Tuple[List[str], List[str]]:
    """
    Export a PyTorch model to an ONNX Intermediate Representation (IR)

    Args:
        preprocessor: ([`PreTrainedTokenizer`], [`FeatureExtractionMixin`] or [`ProcessorMixin`]):
            The preprocessor used for encoding the data.
        model ([`PreTrainedModel`]):
            The model to export.
        config ([`~onnx.config.OnnxConfig`]):
            The ONNX configuration associated with the exported model.
        opset (`int`):
            The version of the ONNX operator set to use.
        output (`Path`):
            Directory to store the exported ONNX model.
        device (`str`, *optional*, defaults to `cpu`):
            The device on which the ONNX model will be exported. Either `cpu` or `cuda`.

    Returns:
        `Tuple[List[str], List[str]]`: A tuple with an ordered list of the model's inputs, and the named inputs from
        the ONNX configuration.
    """

    if isinstance(preprocessor, PreTrainedTokenizerBase) and tokenizer is not None:
        raise ValueError("You cannot provide both a tokenizer and a preprocessor to export the model.")
    if tokenizer is not None:
        warnings.warn(
            "The `tokenizer` argument is deprecated and will be removed in version 5 of Transformers. Use"
            " `preprocessor` instead.",
            FutureWarning,
        )
        logger.info("Overwriting the `preprocessor` argument with `tokenizer` to generate dummmy inputs.")
        preprocessor = tokenizer

    if issubclass(type(model), PreTrainedModel):
        import torch
        from torch.onnx import export as onnx_export

        logger.info(f"Using framework PyTorch: {torch.__version__}")
        with torch.no_grad():
            model.config.return_dict = True
            model.eval()

            # Check if we need to override certain configuration item
            if config.values_override is not None:
                logger.info(f"Overriding {len(config.values_override)} configuration item(s)")
                for override_config_key, override_config_value in config.values_override.items():
                    logger.info(f"\t- {override_config_key} -> {override_config_value}")
                    setattr(model.config, override_config_key, override_config_value)

            # Ensure inputs match
            # TODO: Check when exporting QA we provide "is_pair=True"
            model_inputs = config.generate_dummy_inputs(preprocessor, framework=TensorType.PYTORCH)
            device = torch.device(device)
            if device.type == "cuda" and torch.cuda.is_available():
                model.to(device)
                model_inputs_device = {}
                for k, v in model_inputs.items():
                    if isinstance(v, Tuple):
                        model_inputs_device[k] = tuple(
                            x.to(device) if isinstance(x, torch.Tensor) else None for x in v
                        )
                    elif isinstance(v, List):
                        model_inputs_device[k] = [
                            tuple(x.to(device) if isinstance(x, torch.Tensor) else None for x in t) for t in v
                        ]
                    else:
                        model_inputs_device[k] = v.to(device)

                model_inputs = model_inputs_device

            inputs_match, matched_inputs = ensure_model_and_config_inputs_match(model, model_inputs.keys())
            onnx_outputs = list(config.outputs.keys())

            if not inputs_match:
                raise ValueError("Model and config inputs doesn't match")

            config.patch_ops()

            onnx_export(
                model,
                (model_inputs,),
                f=output.as_posix(),
                input_names=list(config.inputs.keys()),
                output_names=onnx_outputs,
                dynamic_axes=dict(chain(config.inputs.items(), config.outputs.items())),
                do_constant_folding=True,
                opset_version=opset,
            )

            config.restore_ops()



SUMMARY:
**PyTorch Model Export Function Summary**

### Purpose and Functionality

The `export_pytorch` function exports a PyTorch model to an ONNX Intermediate Representation (IR) file. It takes in various parameters such as the preprocessor, model, configuration, opset version, output directory, device, and optional tokenizer and device.

**Key Implementation Details**

1.  **Preprocessing**: The function preprocesses the input data using the provided `preprocessor` or `tokenizer`, and checks if both are provided.
2.  **Model Evaluation**: It evaluates the model in evaluation mode and sets the configuration to return a dictionary (`model.config.return_dict = True`).
3.  **Configuration Patching**: The function patches ONNX operations based on the provided configurations (`config.patch_ops()`).
4.  **ONNX Export**: It exports the model using PyTorch's `onnx_export` function, specifying the input and output names, opset version, and other settings.
5.  **Input Validation**: It checks if the model inputs match the configuration inputs and raises a `ValueError` if they do not.

**Noteable Dependencies and Requirements**

1.  The function uses PyTorch as its underlying framework (`torch`).
2.  It relies on the ONNX library (`torch.onnx`) for exporting to ONNX.
3.  External dependencies include:
    *   `Path` from the ` pathlib` module (not explicitly used in this excerpt but implied)
    *   `Logger`, `warnings`, and `FutureWarning` from the PyTorch documentation
    *   `TensorType` (enum) from the ONNX documentation

### Return Values

Upon successful export, the function returns a tuple containing:

1.  An ordered list of the model's inputs (`matched_inputs`)
2.  A list of named outputs from the ONNX configuration (`onnx_outputs`)


key: chroma:document
string_value:
CODE:

def export_tensorflow(
    preprocessor: Union["PreTrainedTokenizer", "FeatureExtractionMixin"],
    model: "TFPreTrainedModel",
    config: OnnxConfig,
    opset: int,
    output: Path,
    tokenizer: "PreTrainedTokenizer" = None,
) -> Tuple[List[str], List[str]]:
    """
    Export a TensorFlow model to an ONNX Intermediate Representation (IR)

    Args:
        preprocessor: ([`PreTrainedTokenizer`] or [`FeatureExtractionMixin`]):
            The preprocessor used for encoding the data.
        model ([`TFPreTrainedModel`]):
            The model to export.
        config ([`~onnx.config.OnnxConfig`]):
            The ONNX configuration associated with the exported model.
        opset (`int`):
            The version of the ONNX operator set to use.
        output (`Path`):
            Directory to store the exported ONNX model.

    Returns:
        `Tuple[List[str], List[str]]`: A tuple with an ordered list of the model's inputs, and the named inputs from
        the ONNX configuration.
    """
    import onnx
    import tensorflow as tf
    import tf2onnx

    if isinstance(preprocessor, PreTrainedTokenizerBase) and tokenizer is not None:
        raise ValueError("You cannot provide both a tokenizer and preprocessor to export the model.")
    if tokenizer is not None:
        warnings.warn(
            "The `tokenizer` argument is deprecated and will be removed in version 5 of Transformers. Use"
            " `preprocessor` instead.",
            FutureWarning,
        )
        logger.info("Overwriting the `preprocessor` argument with `tokenizer` to generate dummmy inputs.")
        preprocessor = tokenizer

    model.config.return_dict = True

    # Check if we need to override certain configuration item
    if config.values_override is not None:
        logger.info(f"Overriding {len(config.values_override)} configuration item(s)")
        for override_config_key, override_config_value in config.values_override.items():
            logger.info(f"\t- {override_config_key} -> {override_config_value}")
            setattr(model.config, override_config_key, override_config_value)

    # Ensure inputs match
    model_inputs = config.generate_dummy_inputs(preprocessor, framework=TensorType.TENSORFLOW)
    inputs_match, matched_inputs = ensure_model_and_config_inputs_match(model, model_inputs.keys())
    onnx_outputs = list(config.outputs.keys())

    input_signature = [
        tf.TensorSpec([None] * tensor.ndim, dtype=tensor.dtype, name=key) for key, tensor in model_inputs.items()
    ]
    onnx_model, _ = tf2onnx.convert.from_keras(model, input_signature, opset=opset)
    onnx.save(onnx_model, output.as_posix())
    config.restore_ops()



SUMMARY:
**Summary:**

`export_tensorflow` function exports a TensorFlow model to an ONNX IR.

**Main Purpose and Functionality:**

* Takes in a model, preprocessor, configuration, and output directory
* Converts the model to ONNX format with specified opset version
* Generates dummy inputs for the model if necessary
* Returns ordered lists of model inputs and named inputs from configuration

**Key Implementation Details:**

* Uses `tf2onnx` library to convert Keras model to ONNX format
* Modifies model configuration with override values if provided
* Ensures input signatures match between model and configuration

**Notable Dependencies and Requirements:**

* `onnx` and `tensorflow` libraries
* `tf2onnx` library for model conversion
* `PreTrainedTokenizer`, `TFPreTrainedModel`, and `OnnxConfig` classes from Hugging Face Transformers library


key: chroma:document
string_value:
CODE:

def export(
    preprocessor: Union["PreTrainedTokenizer", "FeatureExtractionMixin", "ProcessorMixin"],
    model: Union["PreTrainedModel", "TFPreTrainedModel"],
    config: OnnxConfig,
    opset: int,
    output: Path,
    tokenizer: "PreTrainedTokenizer" = None,
    device: str = "cpu",
) -> Tuple[List[str], List[str]]:
    """
    Export a Pytorch or TensorFlow model to an ONNX Intermediate Representation (IR)

    Args:
        preprocessor: ([`PreTrainedTokenizer`], [`FeatureExtractionMixin`] or [`ProcessorMixin`]):
            The preprocessor used for encoding the data.
        model ([`PreTrainedModel`] or [`TFPreTrainedModel`]):
            The model to export.
        config ([`~onnx.config.OnnxConfig`]):
            The ONNX configuration associated with the exported model.
        opset (`int`):
            The version of the ONNX operator set to use.
        output (`Path`):
            Directory to store the exported ONNX model.
        device (`str`, *optional*, defaults to `cpu`):
            The device on which the ONNX model will be exported. Either `cpu` or `cuda`. Only PyTorch is supported for
            export on CUDA devices.

    Returns:
        `Tuple[List[str], List[str]]`: A tuple with an ordered list of the model's inputs, and the named inputs from
        the ONNX configuration.
    """
    if not (is_torch_available() or is_tf_available()):
        raise ImportError(
            "Cannot convert because neither PyTorch nor TensorFlow are not installed. "
            "Please install torch or tensorflow first."
        )

    if is_tf_available() and isinstance(model, TFPreTrainedModel) and device == "cuda":
        raise RuntimeError("`tf2onnx` does not support export on CUDA device.")

    if isinstance(preprocessor, PreTrainedTokenizerBase) and tokenizer is not None:
        raise ValueError("You cannot provide both a tokenizer and a preprocessor to export the model.")
    if tokenizer is not None:
        warnings.warn(
            "The `tokenizer` argument is deprecated and will be removed in version 5 of Transformers. Use"
            " `preprocessor` instead.",
            FutureWarning,
        )
        logger.info("Overwriting the `preprocessor` argument with `tokenizer` to generate dummmy inputs.")
        preprocessor = tokenizer

    if is_torch_available():
        from ..utils import get_torch_version

        if not config.is_torch_support_available:
            logger.warning(
                f"Unsupported PyTorch version for this model. Minimum required is {config.torch_onnx_minimum_version},"
                f" got: {get_torch_version()}"
            )

    if is_torch_available() and issubclass(type(model), PreTrainedModel):
        return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer, device=device)
    elif is_tf_available() and issubclass(type(model), TFPreTrainedModel):


SUMMARY:
**Summary:**

This function, `export`, exports a PyTorch or TensorFlow model to an ONNX Intermediate Representation (IR) with specified parameters.

**Main Purpose and Functionality:**
The primary purpose of this function is to convert a trained PyTorch or TensorFlow model into an ONNX format, which can be used for inference on other platforms. The function takes various inputs, including the preprocessor, model, configuration, operator set version, output directory, device, and tokenizer (optional).

**Key Implementation Details:**

* It checks availability of required libraries (PyTorch and TensorFlow) and raises errors if not installed.
* It handles deprecated `tokenizer` argument and logs a warning if it is provided.
* For PyTorch models, it uses the `export_pytorch` function to export the model. This function is not shown in the provided code chunk.
* For TensorFlow models, it uses the `export_tensorflow` function to export the model. This function is also not shown in the provided code chunk.

**Notable Dependencies or Requirements:**

* Requires PyTorch and/or TensorFlow libraries
* Uses `PreTrainedTokenizer`, `FeatureExtractionMixin`, `ProcessorMixin`, `PreTrainedModel`, `TFPreTrainedModel`, `OnnxConfig` classes
* Exports to ONNX format with specified parameters

**Method Signatures and Interfaces:**

* Not shown in the provided code chunk, but likely includes `export_pytorch` and `export_tensorflow` functions.


key: chroma:document
string_value:
CODE:

def validate_model_outputs(
    config: OnnxConfig,
    preprocessor: Union["PreTrainedTokenizer", "FeatureExtractionMixin", "ProcessorMixin"],
    reference_model: Union["PreTrainedModel", "TFPreTrainedModel"],
    onnx_model: Path,
    onnx_named_outputs: List[str],
    atol: float,
    tokenizer: "PreTrainedTokenizer" = None,
):
    from onnxruntime import InferenceSession, SessionOptions

    logger.info("Validating ONNX model...")

    if isinstance(preprocessor, PreTrainedTokenizerBase) and tokenizer is not None:
        raise ValueError("You cannot provide both a tokenizer and a preprocessor to validate the model outputs.")
    if tokenizer is not None:
        warnings.warn(
            "The `tokenizer` argument is deprecated and will be removed in version 5 of Transformers. Use"
            " `preprocessor` instead.",
            FutureWarning,
        )
        logger.info("Overwriting the `preprocessor` argument with `tokenizer` to generate dummmy inputs.")
        preprocessor = tokenizer

    # generate inputs with a different batch_size and seq_len that was used for conversion to properly test
    # dynamic input shapes.
    if is_torch_available() and issubclass(type(reference_model), PreTrainedModel):
        reference_model_inputs = config.generate_dummy_inputs(
            preprocessor,
            batch_size=config.default_fixed_batch + 1,
            seq_length=config.default_fixed_sequence + 1,
            framework=TensorType.PYTORCH,
        )
    else:
        reference_model_inputs = config.generate_dummy_inputs(
            preprocessor,
            batch_size=config.default_fixed_batch + 1,
            seq_length=config.default_fixed_sequence + 1,
            framework=TensorType.TENSORFLOW,
        )

    # Create ONNX Runtime session
    options = SessionOptions()
    session = InferenceSession(onnx_model.as_posix(), options, providers=["CPUExecutionProvider"])

    # Compute outputs from the reference model
    if is_torch_available() and issubclass(type(reference_model), PreTrainedModel):
        reference_model.to("cpu")
    ref_outputs = reference_model(**reference_model_inputs)
    ref_outputs_dict = {}

    # We flatten potential collection of outputs (i.e. past_keys) to a flat structure
    for name, value in ref_outputs.items():
        # Overwriting the output name as "present" since it is the name used for the ONNX outputs
        # ("past_key_values" being taken for the ONNX inputs)
        if name == "past_key_values":
            name = "present"
        if isinstance(value, (list, tuple)):
            value = config.flatten_output_collection_property(name, value)
            ref_outputs_dict.update(value)
        else:
            ref_outputs_dict[name] = value

    # Create onnxruntime inputs from the reference model inputs
    reference_model_inputs_onnxruntime = config.generate_dummy_inputs_onnxruntime(reference_model_inputs)

    # We flatten potential collection of inputs (i.e. past_keys)
    onnx_inputs = {}
    for name, value in reference_model_inputs_onnxruntime.items():
        if isinstance(value, (list, tuple)):
            value = config.flatten_output_collection_property(name, value)
            onnx_inputs.update({tensor_name: pt_tensor.numpy() for tensor_name, pt_tensor in value.items()})
        else:
            onnx_inputs[name] = value.numpy()

    # Compute outputs from the ONNX model
    onnx_outputs = session.run(onnx_named_outputs, onnx_inputs)

    # Check we have a subset of the keys into onnx_outputs against ref_outputs
    ref_outputs_set, onnx_outputs_set = set(ref_outputs_dict.keys()), set(onnx_named_outputs)
    if not onnx_outputs_set.issubset(ref_outputs_set):
        logger.info(
            f"\t-[x] ONNX model output names {onnx_outputs_set} do not match reference model {ref_outputs_set}"
        )

        raise ValueError(
            "Outputs doesn't match between reference model and ONNX exported model: "
            f"{onnx_outputs_set.difference(ref_outputs_set)}"
        )
    else:
        logger.info(f"\t-[✓] ONNX model output names match reference model ({onnx_outputs_set})")

    # Check the shape and values match
    for name, ort_value in zip(onnx_named_outputs, onnx_outputs):
        if is_torch_available() and issubclass(type(reference_model), PreTrainedModel):
            ref_value = ref_outputs_dict[name].detach().numpy()
        else:
            ref_value = ref_outputs_dict[name].numpy()
        logger.info(f'\t- Validating ONNX Model output "{name}":')

        # Shape
        if not ort_value.shape == ref_value.shape:
            logger.info(f"\t\t-[x] shape {ort_value.shape} doesn't match {ref_value.shape}")
            raise ValueError(
                "Outputs shape doesn't match between reference model and ONNX exported model: "
                f"Got {ref_value.shape} (reference) and {ort_value.shape} (ONNX)"
            )
        else:
            logger.info(f"\t\t-[✓] {ort_value.shape} matches {ref_value.shape}")

        # Values
        if not np.allclose(ref_value, ort_value, atol=atol):
            bad_indices = np.logical_not(np.isclose(ref_value, ort_value, atol=atol))
            logger.info(f"\t\t-[x] values not close enough (atol: {atol})")
            raise ValueError(
                "Outputs values doesn't match between reference model and ONNX exported model: "
                f"Got max absolute difference of: {np.amax(np.abs(ref_value - ort_value))} for "
                f"{ref_value[bad_indices]} vs {ort_value[bad_indices]}"
            )
        else:


SUMMARY:
**Function Summary**

`validate_model_outputs`

Validates the outputs of an ONNX model against a reference TPU model. The function takes in a configuration, preprocessor, and onnx model as input.

**Primary Purpose**: To ensure that the output of the ONNX model matches the expected output of the reference TPU model.

**Key Implementation Details**

1. **Reference Model Input Generation**: Dynamic inputs are generated for the reference model to test dynamic shapes.
2. **ONNX Session Creation**: An INferenceSession is created with the onnx model and session options.
3. **Reference Model Output Extraction**: Outputs from the reference model are extracted using the `to` method.
4. **FLatten Collection Properties**: Potential collection of outputs (e.g., "past_key_values") is flattened to a flat structure.

**Notable Dependencies/Requirements**

1. `onnxruntime`: Used for creating an INferenceSession.
2. `PreTrainedTokenizer` and `PreTrainedModel`: Used as input preprocessor and reference model, respectively.

**Check Points**

1. Output names from the ONNX model must match the expected output names of the reference TPU model.
2. Output shapes and values from both models must match within a certain tolerance (`atol`).


key: chroma:document
string_value:
CODE:

def ensure_model_and_config_inputs_match(
    model: Union["PreTrainedModel", "TFPreTrainedModel"], model_inputs: Iterable[str]
) -> Tuple[bool, List[str]]:
    """

    :param model_inputs: :param config_inputs: :return:
    """
    if is_torch_available() and issubclass(type(model), PreTrainedModel):
        forward_parameters = signature(model.forward).parameters
    else:
        forward_parameters = signature(model.call).parameters
    model_inputs_set = set(model_inputs)

    # We are fine if config_inputs has more keys than model_inputs
    forward_inputs_set = set(forward_parameters.keys())
    is_ok = model_inputs_set.issubset(forward_inputs_set)

    # Make sure the input order match (VERY IMPORTANT !!!!)
    matching_inputs = forward_inputs_set.intersection(model_inputs_set)
    ordered_inputs = [parameter for parameter in forward_parameters.keys() if parameter in matching_inputs]


SUMMARY:
**Summary**

`ensure_model_and_config_inputs_match` function ensures that input parameters match between model and configuration inputs.

### Purpose and Functionality

Verifies whether input parameters of a pre-trained model (`model`) align with the specified model inputs (`model_inputs`). Returns a boolean indicating success and a list of matched input parameters in order.

### Key Implementation Details

* Checks if `model` is compatible with both TensorFlow and PyTorch models.
* Extracts forward function parameters using `signature(model.forward).parameters` (for PyTorch) or `signature(model.call).parameters` (for TensorFlow).
* Compares input sets between model inputs and forward function parameters.

### Notable Dependencies and Requirements

* `PyTorch` or `TensorFlow` models (`PreTrainedModel`, `TFPreTrainedModel`)
* `Union` type hint for model parameter type
* `Iterable[str]` for model inputs
* `Signature` object from `torch.nn.Module` (for PyTorch) or `tf.keras.Model` (for TensorFlow)
* `is_torch_available()` function (optional)

### Return Value

* `bool`: whether input parameters match
* `List[str]`: list of matched input parameters in order


key: chroma:document
string_value:
CODE:

class ParameterFormat(Enum):
    Float = c_float

    @property
    def size(self) -> int:
        """
        Number of byte required for this data type

        Returns:
            Integer > 0
        """


SUMMARY:
**Summary**

The `ParameterFormat` class defines an enumeration with a single value `Float`, which is mapped to `c_float`. The class has a read-only property `size` that returns the size of data required for this data type in bytes. 

* **Main purpose and functionality**: Provides a way to determine data size requirements for different parameter formats.
* **Key implementation details**:
	+ Use of `sizeof` function to calculate data size.
	+ Read-only property `size` with return type `int`.
* **Notable dependencies or requirements**:
	+ `c_float` (a C-style float) enum value, which suggests usage in a low-level programming context possibly involving C or Cython.


key: chroma:document
string_value:
CODE:

def compute_effective_axis_dimension(dimension: int, fixed_dimension: int, num_token_to_add: int = 0) -> int:
    """

    Args:
        dimension:
        fixed_dimension:
        num_token_to_add:

    Returns:

    """
    # < 0 is possible if using a dynamic axis
    if dimension <= 0:
        dimension = fixed_dimension

    dimension -= num_token_to_add


SUMMARY:
**Code Summary**

`compute_effective_axis.Dimension` function:

### Purpose and Functionality

Compute the effective dimension of a feature axis by subtracting token additions.

```python
def compute_effective_axis_dimension(dimension: int, fixed_dimension: int, num_token_to_add: int = 0) -> int:
    """
    Compute the effective dimension of a feature axis.
    
    Args:
        dimension (int): Initial dimension value.
        fixed_dimension (int): Fixed dimension value.
        num_token_to_add (int): Number of tokens to add. Defaults to 0.
    
    Returns:
        int: Effective dimension after subtraction.
    """
```

### Key Implementation Details

*   Subtracts `num_token_to_add` from the input `dimension`, using `fixed_dimension` as a fallback if `dimension` is less than or equal to 0.

```python
if dimension <= 0:
    dimension = fixed_dimension
dimension -= num_token_to_add
```

### Notable Dependencies or Requirements

-   None explicitly stated, but can be inferred to work within the context of feature axis dimensions.
-   Depends on Python's type hinting system for parameter and return type validation.

### Further Analysis

This function appears to be part of a data processing pipeline, likely used in natural language processing (NLP) tasks where token additions are necessary. The `num_token_to_add` parameter allows developers to adjust this behavior dynamically.


key: chroma:document
string_value:
CODE:

def compute_serialized_parameters_size(num_parameters: int, dtype: ParameterFormat) -> int:
    """
    Compute the size taken by all the parameters in the given the storage format when serializing the model

    Args:
        num_parameters: Number of parameters to be saved
        dtype: The data format each parameter will be saved

    Returns:
        Size (in byte) taken to save all the parameters
    """


SUMMARY:
**Summary**

```python
compute_serialized_parameters_size(num_parameters: int, dtype: ParameterFormat) -> int:
  # Compute size of serialized model parameters based on data type
  sizes = num_parameters * [ParameterFormat.ANY, ParameterFormat.FloatingPoint].index(dtype) * 8
  return sizes
```

**Key Insights**

* Calculates the total size of model parameters in a given storage format.
* Uses `num_parameters` and `dtype` to determine the computation.
* Notable use of `[ParameterFormat.ANY, ParameterFormat.FloatingPoint].index(dtype)` to map `dtype` to a byte size multiplier.


key: chroma:document
string_value:
CODE:

def get_preprocessor(model_name: str) -> Optional[Union["AutoTokenizer", "AutoFeatureExtractor", "AutoProcessor"]]:
    """
    Gets a preprocessor (tokenizer, feature extractor or processor) that is available for `model_name`.

    Args:
        model_name (`str`): Name of the model for which a preprocessor are loaded.

    Returns:
        `Optional[Union[AutoTokenizer, AutoFeatureExtractor, AutoProcessor]]`:
            If a processor is found, it is returned. Otherwise, if a tokenizer or a feature extractor exists, it is
            returned. If both a tokenizer and a feature extractor exist, an error is raised. The function returns
            `None` if no preprocessor is found.
    """
    # Avoid circular imports by only importing this here.
    from .. import AutoFeatureExtractor, AutoProcessor, AutoTokenizer  # tests_ignore

    try:
        return AutoProcessor.from_pretrained(model_name)
    except (ValueError, OSError, KeyError):
        tokenizer, feature_extractor = None, None
        try:
            tokenizer = AutoTokenizer.from_pretrained(model_name)
        except (OSError, KeyError):
            pass
        try:
            feature_extractor = AutoFeatureExtractor.from_pretrained(model_name)
        except (OSError, KeyError):
            pass

        if tokenizer is not None and feature_extractor is not None:
            raise ValueError(
                f"Couldn't auto-detect preprocessor for {model_name}. Found both a tokenizer and a feature extractor."
            )
        elif tokenizer is None and feature_extractor is None:
            return None
        elif tokenizer is not None:
            return tokenizer
        else:


SUMMARY:
**Code Summary: `get_preprocessor` function**

This function retrieves a preprocessor (tokenizer, feature extractor, or processor) for a given model name.

**Primary Purpose:** Returns a preprocessor for the specified model name.

**Key Implementation Details:**

*   Attempts to retrieve an `AutoProcessor` directly from pre-trained models.
*   Falls back to retrieving a `Tokenizer` and then a `FeatureExtractor` if both exist or just a Tokenizer/FeatureExtractor.

**Method Signatures and Interfaces:**

*   `from_pretrained(model_name)`: Retrieves the preprocessor for the specified model name.
*   `AutoProcessor.from_pretrained(model_name)`:
    *   Returns an instance of `AutoProcessor`.
    *   Raises `ValueError` if not found (model name may be invalid).
    *   Raises `OSError` if failed to retrieve the processor.
    *   Raises `KeyError` if model name is unknown.

**Notable Dependencies and Requirements:**

*   `AutoTokenizer`, `AutoFeatureExtractor`, and `AutoProcessor`: Preprocessor classes for tokenization, feature extraction, and processing respectively.


key: chroma:document
string_value:
CODE:

def export_with_optimum(args):
    if is_optimum_available():
        from optimum.version import __version__ as optimum_version

        parsed_optimum_version = version.parse(optimum_version)
        if parsed_optimum_version < version.parse(MIN_OPTIMUM_VERSION):
            raise RuntimeError(
                f"transformers.onnx requires optimum >= {MIN_OPTIMUM_VERSION} but {optimum_version} is installed. You "
                "can upgrade optimum by running: pip install -U optimum[exporters]"
            )
    else:
        raise RuntimeError(
            "transformers.onnx requires optimum to run, you can install the library by running: pip install "
            "optimum[exporters]"
        )
    cmd_line = [
        sys.executable,
        "-m",
        "optimum.exporters.onnx",
        f"--model {args.model}",
        f"--task {args.feature}",
        f"--framework {args.framework}" if args.framework is not None else "",
        f"{args.output}",
    ]
    proc = subprocess.Popen(cmd_line, stdout=subprocess.PIPE)
    proc.wait()

    logger.info(
        "The export was done by optimum.exporters.onnx. We recommend using to use this package directly in future, as "
        "transformers.onnx is deprecated, and will be removed in v5. You can find more information here: "
        "https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model."


SUMMARY:
**Summary**

The `export_with_optimum` function exports an ONNX model with optimal functionality using the Optimum library. It:

*   Checks if Optimum version >= MIN_OPTIMUM_VERSION is available and raises an error if not.
*   Configures a subprocess to run Optimum's exporters module with the provided model, feature, framework, and output options.
*   Waits for the subprocess to complete and logs an informational message.

**Key Implementation Details**

*   The function takes in `args.model`, `args.feature`, `args.framework`, and `args.output` as command-line arguments.
*   It uses `subprocess.Popen` to run Optimum's exporters module with the specified options.
*   `optimum_version.parse` is used to validate the Optimum version, ensuring it meets the minimum requirement.

**Notable Dependencies or Requirements**

*   The Optimum library (`optimum[exporters]`) and its specific version (>= MIN_OPTIMUM_VERSION).
*   No additional external dependencies.


key: chroma:document
string_value:
CODE:

def export_with_transformers(args):
    args.output = args.output if args.output.is_file() else args.output.joinpath("model.onnx")
    if not args.output.parent.exists():
        args.output.parent.mkdir(parents=True)

    # Allocate the model
    model = FeaturesManager.get_model_from_feature(
        args.feature, args.model, framework=args.framework, cache_dir=args.cache_dir
    )

    model_kind, model_onnx_config = FeaturesManager.check_supported_model_or_raise(model, feature=args.feature)
    onnx_config = model_onnx_config(model.config)

    if model_kind in ENCODER_DECODER_MODELS:
        encoder_model = model.get_encoder()
        decoder_model = model.get_decoder()

        encoder_onnx_config = onnx_config.get_encoder_config(encoder_model.config)
        decoder_onnx_config = onnx_config.get_decoder_config(
            encoder_model.config, decoder_model.config, feature=args.feature
        )

        if args.opset is None:
            args.opset = max(encoder_onnx_config.default_onnx_opset, decoder_onnx_config.default_onnx_opset)

        if args.opset < min(encoder_onnx_config.default_onnx_opset, decoder_onnx_config.default_onnx_opset):
            raise ValueError(
                f"Opset {args.opset} is not sufficient to export {model_kind}. At least "
                f" {min(encoder_onnx_config.default_onnx_opset, decoder_onnx_config.default_onnx_opset)} is required."
            )

        preprocessor = AutoFeatureExtractor.from_pretrained(args.model)

        onnx_inputs, onnx_outputs = export(
            preprocessor,
            encoder_model,
            encoder_onnx_config,
            args.opset,
            args.output.parent.joinpath("encoder_model.onnx"),
        )

        validate_model_outputs(
            encoder_onnx_config,
            preprocessor,
            encoder_model,
            args.output.parent.joinpath("encoder_model.onnx"),
            onnx_outputs,
            args.atol if args.atol else encoder_onnx_config.atol_for_validation,
        )

        preprocessor = AutoTokenizer.from_pretrained(args.model)

        onnx_inputs, onnx_outputs = export(
            preprocessor,
            decoder_model,
            decoder_onnx_config,
            args.opset,
            args.output.parent.joinpath("decoder_model.onnx"),
        )

        validate_model_outputs(
            decoder_onnx_config,
            preprocessor,
            decoder_model,
            args.output.parent.joinpath("decoder_model.onnx"),
            onnx_outputs,
            args.atol if args.atol else decoder_onnx_config.atol_for_validation,
        )
        logger.info(
            f"All good, model saved at: {args.output.parent.joinpath('encoder_model.onnx').as_posix()},"
            f" {args.output.parent.joinpath('decoder_model.onnx').as_posix()}"
        )

    else:
        # Instantiate the appropriate preprocessor
        if args.preprocessor == "auto":
            preprocessor = get_preprocessor(args.model)
        elif args.preprocessor == "tokenizer":
            preprocessor = AutoTokenizer.from_pretrained(args.model)
        elif args.preprocessor == "image_processor":
            preprocessor = AutoImageProcessor.from_pretrained(args.model)
        elif args.preprocessor == "feature_extractor":
            preprocessor = AutoFeatureExtractor.from_pretrained(args.model)
        elif args.preprocessor == "processor":
            preprocessor = AutoProcessor.from_pretrained(args.model)
        else:
            raise ValueError(f"Unknown preprocessor type '{args.preprocessor}'")

        # Ensure the requested opset is sufficient
        if args.opset is None:
            args.opset = onnx_config.default_onnx_opset

        if args.opset < onnx_config.default_onnx_opset:
            raise ValueError(
                f"Opset {args.opset} is not sufficient to export {model_kind}. "
                f"At least  {onnx_config.default_onnx_opset} is required."
            )

        onnx_inputs, onnx_outputs = export(
            preprocessor,
            model,
            onnx_config,
            args.opset,
            args.output,
        )

        if args.atol is None:
            args.atol = onnx_config.atol_for_validation

        validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
        logger.info(f"All good, model saved at: {args.output.as_posix()}")
        warnings.warn(
            "The export was done by transformers.onnx which is deprecated and will be removed in v5. We recommend"
            " using optimum.exporters.onnx in future. You can find more information here:"
            " https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model.",
            FutureWarning,


SUMMARY:
**Function Summary: `export_with_transformers`**

The `export_with_transformers` function exports a pre-trained model to ONNX format with optional transformations. The primary purpose is to save the model in a specified output directory.

**Key Implementation Details:**

* Allocate and prepare the model for export using `FeaturesManager`.
* Extract or create a preprocessor based on the specified `preprocessor` type.
* Validate the model outputs and set atol tolerance if required.
* Export the model to ONNX format with specified opsets and validation thresholds.

**Method Signatures and Interfaces:**

* `export(preprocessor, model, onnx_config, opset, output)`: Exports the preprocessed input data to ONNX format.
* `validate_model_outputs(onnx_config, preprocessor, model, output, onnx_outputs, atol)`: Validates the model outputs and sets atol tolerance if required.

**Notable Dependencies and Requirements:**

* `ONNXConfig` interface
* `AutoFeatureExtractor` and `AutoTokenizer` classes
* `export` function with options for opset and atol
* `validator` class (not shown in the code snippet)

**Input Parameters:**

* `args`: An object containing:
	+ `output`: The output directory path.
	+ `feature`: The feature type of the model.
	+ `model`: The pre-trained model object.
	+ `framework`: The framework for the model.
	+ `cache_dir`: The cache directory path.
	+ `opset`: The minimum opset version required for export (required for decoder models).
	+ `preprocessor`: The type of transformation to apply to the input data.
	+ `atol`: The absolute tolerance value for validation.

**Return Values:**

* None (the exported model is saved in the specified output directory).


key: chroma:document
string_value:
CODE:

def main():
    parser = ArgumentParser("Hugging Face Transformers ONNX exporter")
    parser.add_argument(
        "-m", "--model", type=str, required=True, help="Model ID on huggingface.co or path on disk to load model from."
    )
    parser.add_argument(
        "--feature",
        default="default",
        help="The type of features to export the model with.",
    )
    parser.add_argument("--opset", type=int, default=None, help="ONNX opset version to export the model with.")
    parser.add_argument(
        "--atol", type=float, default=None, help="Absolute difference tolerance when validating the model."
    )
    parser.add_argument(
        "--framework",
        type=str,
        choices=["pt", "tf"],
        default=None,
        help=(
            "The framework to use for the ONNX export."
            " If not provided, will attempt to use the local checkpoint's original framework"
            " or what is available in the environment."
        ),
    )
    parser.add_argument("output", type=Path, help="Path indicating where to store generated ONNX model.")
    parser.add_argument("--cache_dir", type=str, default=None, help="Path indicating where to store cache.")
    parser.add_argument(
        "--preprocessor",
        type=str,
        choices=["auto", "tokenizer", "feature_extractor", "image_processor", "processor"],
        default="auto",
        help="Which type of preprocessor to use. 'auto' tries to automatically detect it.",
    )
    parser.add_argument(
        "--export_with_transformers",
        action="store_true",
        help=(
            "Whether to use transformers.onnx instead of optimum.exporters.onnx to perform the ONNX export. It can be "
            "useful when exporting a model supported in transformers but not in optimum, otherwise it is not "
            "recommended."
        ),
    )

    args = parser.parse_args()
    if args.export_with_transformers or not is_optimum_available():
        export_with_transformers(args)
    else:


SUMMARY:
**Function Summary:**

This `main` function serves as the entry point for exporting Hugging Face Transformers models to ONNX format. It uses `optimum.exporters.onnx` and/or `transformers.onnx` exporters, depending on user-defined options.

**Key Functionality:**

1. Parses command-line arguments using `ArgumentParser`, allowing users to specify:
	* Model ID or path
	* Export features type (default: "default")
	* ONNX opset version (optional)
	* Absolute difference tolerance for model validation (optional)
	* Framework to use (default: auto-detected)
2. Checks if `export_with_transformers` option is enabled and attempts to export with `transformers.onnx` exporter if `optimum.exporters.onnx` is not available.
3. Parses output path and cache directory using `Path`.

**Notable Implementation Details:**

1. Uses `is_optimum_available()` function to check for availability of `optimum.exporters.onnx`.
2. Defines several preprocessor types (`auto`, `tokenizer`, `feature_extractor`, etc.) with corresponding argument names.

**External Dependencies:**

* `ArgumentParser`
* `Path`
* `is_optimum_available()` (function)
* `export_with_transformers` and `export_with_optimum` functions or classes are called based on user-defined options, but their implementation is not shown in this code snippet.


key: chroma:document
string_value:
CODE:

class OnnxConfig(ABC):
    """
    Base class for ONNX exportable model describing metadata on how to export the model through the ONNX format.
    """

    default_fixed_batch = 2
    default_fixed_sequence = 8
    default_fixed_num_choices = 4
    torch_onnx_minimum_version = version.parse("1.8")
    _tasks_to_common_outputs = {
        "causal-lm": OrderedDict({"logits": {0: "batch", 1: "sequence"}}),
        "default": OrderedDict({"last_hidden_state": {0: "batch", 1: "sequence"}}),
        "image-classification": OrderedDict({"logits": {0: "batch", 1: "sequence"}}),
        "image-segmentation": OrderedDict(
            {
                "logits": {0: "batch", 1: "sequence"},
                "pred_boxes": {0: "batch", 1: "sequence"},
                "pred_masks": {0: "batch", 1: "sequence"},
            }
        ),
        "masked-im": OrderedDict({"logits": {0: "batch", 1: "sequence"}}),
        "masked-lm": OrderedDict({"logits": {0: "batch", 1: "sequence"}}),
        "multiple-choice": OrderedDict({"logits": {0: "batch"}}),
        "object-detection": OrderedDict(
            {
                "logits": {0: "batch", 1: "sequence"},
                "pred_boxes": {0: "batch", 1: "sequence"},
            }
        ),
        "question-answering": OrderedDict(
            {
                "start_logits": {0: "batch", 1: "sequence"},
                "end_logits": {0: "batch", 1: "sequence"},
            }
        ),
        "semantic-segmentation": OrderedDict({"logits": {0: "batch", 1: "num_labels", 2: "height", 3: "width"}}),
        "seq2seq-lm": OrderedDict({"logits": {0: "batch", 1: "decoder_sequence"}}),
        "sequence-classification": OrderedDict({"logits": {0: "batch"}}),
        "token-classification": OrderedDict({"logits": {0: "batch", 1: "sequence"}}),
        "vision2seq-lm": OrderedDict({"logits": {0: "batch", 1: "sequence"}}),
        "speech2seq-lm": OrderedDict({"logits": {0: "batch", 1: "sequence"}}),
    }

    def __init__(self, config: "PretrainedConfig", task: str = "default", patching_specs: List[PatchingSpec] = None):
        self._config = config

        if task not in self._tasks_to_common_outputs:
            raise ValueError(
                f"{task} is not a supported task, supported tasks: {self._tasks_to_common_outputs.keys()}"
            )
        self.task = task

        self._patching_specs = []
        for spec in patching_specs if patching_specs is not None else []:
            final_spec = spec
            if spec.orig_op is None:
                final_spec = dataclasses.replace(spec, orig_op=getattr(spec.o, spec.name))
            self._patching_specs.append(final_spec)

    @classmethod
    def from_model_config(cls, config: "PretrainedConfig", task: str = "default") -> "OnnxConfig":
        """
        Instantiate a OnnxConfig for a specific model

        Args:
            config: The model's configuration to use when exporting to ONNX

        Returns:
            OnnxConfig for this model
        """
        return cls(config, task=task)

    @property
    @abstractmethod
    def inputs(self) -> Mapping[str, Mapping[int, str]]:
        """
        Mapping containing the axis definition of the input tensors to provide to the model

        Returns:
            For each input: its name associated to the axes symbolic name and the axis position within the tensor
        """
        raise NotImplementedError()

    @property
    def outputs(self) -> Mapping[str, Mapping[int, str]]:
        """
        Mapping containing the axis definition of the output tensors to provide to the model

        Returns:
            For each output: its name associated to the axes symbolic name and the axis position within the tensor
        """
        common_outputs = self._tasks_to_common_outputs[self.task]
        return copy.deepcopy(common_outputs)

    @property
    def values_override(self) -> Optional[Mapping[str, Any]]:
        """
        Dictionary of keys to override in the model's config before exporting

        Returns:
            Dictionary with the keys (and their corresponding values) to override
        """
        if hasattr(self._config, "use_cache"):
            return {"use_cache": False}

        return None

    @property
    def default_batch_size(self) -> int:
        """
        The default batch size to use if no other indication

        Returns:
            Integer > 0
        """
        # Using 2 avoid ONNX making assumption about single sample batch
        return OnnxConfig.default_fixed_batch

    @property
    def default_sequence_length(self) -> int:
        """
        The default sequence length to use if no other indication

        Returns:
            Integer > 0
        """
        return OnnxConfig.default_fixed_sequence

    @property
    def default_num_choices(self) -> int:
        """
        The default number of choices to use if no other indication

        Returns:
            Integer > 0
        """
        return OnnxConfig.default_fixed_num_choices

    @property
    def default_onnx_opset(self) -> int:
        """
        Which onnx opset to use when exporting the model

        Returns:
            Integer ONNX Opset version
        """
        return DEFAULT_ONNX_OPSET

    @property
    def atol_for_validation(self) -> float:
        """
        What absolute tolerance value to use during model conversion validation.

        Returns:
            Float absolute tolerance value.
        """
        return 1e-5

    @property
    def is_torch_support_available(self) -> bool:
        """
        The minimum PyTorch version required to export the model.

        Returns:
            `bool`: Whether the installed version of PyTorch is compatible with the model.
        """
        if is_torch_available():
            from transformers.utils import get_torch_version

            return version.parse(get_torch_version()) >= self.torch_onnx_minimum_version
        else:
            return False

    @staticmethod
    def use_external_data_format(num_parameters: int) -> bool:
        """
        Flag indicating if the model requires using external data format

        Args:
            num_parameters: Number of parameter on the model

        Returns:
            True if model.num_parameters() * size_of(float32) >= 2Gb False otherwise
        """

        return (
            compute_serialized_parameters_size(num_parameters, ParameterFormat.Float)
            >= EXTERNAL_DATA_FORMAT_SIZE_LIMIT
        )

    def _generate_dummy_images(
        self, batch_size: int = 2, num_channels: int = 3, image_height: int = 40, image_width: int = 40
    ):
        images = []
        for _ in range(batch_size):
            data = np.random.rand(image_height, image_width, num_channels) * 255
            images.append(Image.fromarray(data.astype("uint8")).convert("RGB"))
        return images

    def _generate_dummy_audio(
        self, batch_size: int = 2, sampling_rate: int = 22050, time_duration: float = 5.0, frequency: int = 220
    ):
        audio_data = []
        for _ in range(batch_size):
            # time variable
            t = np.linspace(0, time_duration, int(time_duration * sampling_rate), endpoint=False)

            # generate pure sine wave at `frequency` Hz
            audio_data.append(0.5 * np.sin(2 * np.pi * frequency * t))

        return audio_data

    def generate_dummy_inputs(
        self,
        preprocessor: Union["PreTrainedTokenizerBase", "FeatureExtractionMixin", "ImageProcessingMixin"],
        batch_size: int = -1,
        seq_length: int = -1,
        num_choices: int = -1,
        is_pair: bool = False,
        framework: Optional[TensorType] = None,
        num_channels: int = 3,
        image_width: int = 40,
        image_height: int = 40,
        sampling_rate: int = 22050,
        time_duration: float = 5.0,
        frequency: int = 220,
        tokenizer: "PreTrainedTokenizerBase" = None,
    ) -> Mapping[str, Any]:
        """
        Generate inputs to provide to the ONNX exporter for the specific framework

        Args:
            preprocessor: ([`PreTrainedTokenizerBase`], [`FeatureExtractionMixin`], or [`ImageProcessingMixin`]):
                The preprocessor associated with this model configuration.
            batch_size (`int`, *optional*, defaults to -1):
                The batch size to export the model for (-1 means dynamic axis).
            num_choices (`int`, *optional*, defaults to -1):
                The number of candidate answers provided for multiple choice task (-1 means dynamic axis).
            seq_length (`int`, *optional*, defaults to -1):
                The sequence length to export the model for (-1 means dynamic axis).
            is_pair (`bool`, *optional*, defaults to `False`):
                Indicate if the input is a pair (sentence 1, sentence 2)
            framework (`TensorType`, *optional*, defaults to `None`):
                The framework (PyTorch or TensorFlow) that the tokenizer will generate tensors for.
            num_channels (`int`, *optional*, defaults to 3):
                The number of channels of the generated images.
            image_width (`int`, *optional*, defaults to 40):
                The width of the generated images.
            image_height (`int`, *optional*, defaults to 40):
                The height of the generated images.
            sampling_rate (`int`, *optional* defaults to 22050)
                The sampling rate for audio data generation.
            time_duration (`float`, *optional* defaults to 5.0)
                Total seconds of sampling for audio data generation.
            frequency (`int`, *optional* defaults to 220)
                The desired natural frequency of generated audio.

        Returns:
            Mapping[str, Tensor] holding the kwargs to provide to the model's forward function
        """
        from ..feature_extraction_utils import FeatureExtractionMixin
        from ..image_processing_utils import ImageProcessingMixin
        from ..tokenization_utils_base import PreTrainedTokenizerBase

        if isinstance(preprocessor, PreTrainedTokenizerBase) and tokenizer is not None:
            raise ValueError("You cannot provide both a tokenizer and a preprocessor to generate dummy inputs.")
        if tokenizer is not None:
            warnings.warn(
                "The `tokenizer` argument is deprecated and will be removed in version 5 of Transformers. Use"
                " `preprocessor` instead.",
                FutureWarning,
            )
            logger.warning("Overwriting the `preprocessor` argument with `tokenizer` to generate dummmy inputs.")
            preprocessor = tokenizer
        if isinstance(preprocessor, PreTrainedTokenizerBase):
            # If dynamic axis (-1) we forward with a fixed dimension of 2 samples to avoid optimizations made by ONNX
            batch_size = compute_effective_axis_dimension(
                batch_size, fixed_dimension=OnnxConfig.default_fixed_batch, num_token_to_add=0
            )
            # If dynamic axis (-1) we forward with a fixed dimension of 8 tokens to avoid optimizations made by ONNX
            token_to_add = preprocessor.num_special_tokens_to_add(is_pair)
            seq_length = compute_effective_axis_dimension(
                seq_length, fixed_dimension=OnnxConfig.default_fixed_sequence, num_token_to_add=token_to_add
            )
            # Generate dummy inputs according to compute batch and sequence
            input_token = (
                preprocessor.unk_token
                if (preprocessor.unk_token is not None and len(preprocessor.unk_token) > 0)
                else "0"
            )
            dummy_input = [" ".join([input_token]) * seq_length] * batch_size
            if self.task == "multiple-choice":
                # If dynamic axis (-1) we forward with a fixed dimension of 4 candidate answers to avoid optimizations
                # made by ONNX
                num_choices = compute_effective_axis_dimension(
                    num_choices, fixed_dimension=OnnxConfig.default_fixed_num_choices, num_token_to_add=0
                )
                dummy_input = dummy_input * num_choices
                # The shape of the tokenized inputs values is [batch_size * num_choices, seq_length]
                tokenized_input = preprocessor(dummy_input, text_pair=dummy_input)
                # Unflatten the tokenized inputs values expanding it to the shape [batch_size, num_choices, seq_length]
                for k, v in tokenized_input.items():
                    tokenized_input[k] = [v[i : i + num_choices] for i in range(0, len(v), num_choices)]
                return dict(tokenized_input.convert_to_tensors(tensor_type=framework))
            return dict(preprocessor(dummy_input, return_tensors=framework))
        elif isinstance(preprocessor, ImageProcessingMixin):
            if preprocessor.model_input_names[0] != "pixel_values":
                raise ValueError(
                    f"The `preprocessor` is an image processor ({preprocessor.__class__.__name__}) and expects"
                    f' `model_input_names[0]` to be "pixel_values", but got {preprocessor.model_input_names[0]}'
                )
            # If dynamic axis (-1) we forward with a fixed dimension of 2 samples to avoid optimizations made by ONNX
            batch_size = compute_effective_axis_dimension(batch_size, fixed_dimension=OnnxConfig.default_fixed_batch)
            dummy_input = self._generate_dummy_images(batch_size, num_channels, image_height, image_width)
            return dict(preprocessor(images=dummy_input, return_tensors=framework))
        elif isinstance(preprocessor, FeatureExtractionMixin) and preprocessor.model_input_names[0] == "pixel_values":
            # If dynamic axis (-1) we forward with a fixed dimension of 2 samples to avoid optimizations made by ONNX
            batch_size = compute_effective_axis_dimension(batch_size, fixed_dimension=OnnxConfig.default_fixed_batch)
            dummy_input = self._generate_dummy_images(batch_size, num_channels, image_height, image_width)
            return dict(preprocessor(images=dummy_input, return_tensors=framework))
        elif (
            isinstance(preprocessor, FeatureExtractionMixin) and preprocessor.model_input_names[0] == "input_features"
        ):
            # If dynamic axis (-1) we forward with a fixed dimension of 2 samples to avoid optimizations made by ONNX
            batch_size = compute_effective_axis_dimension(batch_size, fixed_dimension=OnnxConfig.default_fixed_batch)
            dummy_input = self._generate_dummy_audio(batch_size, sampling_rate, time_duration, frequency)
            return dict(preprocessor(dummy_input, return_tensors=framework))
        else:
            raise ValueError(
                "Unable to generate dummy inputs for the model. Please provide a tokenizer or a preprocessor."
            )

    def generate_dummy_inputs_onnxruntime(self, reference_model_inputs: Mapping[str, Any]) -> Mapping[str, Any]:
        """
        Generate inputs for ONNX Runtime using the reference model inputs. Override this to run inference with seq2seq
        models which have the encoder and decoder exported as separate ONNX files.

        Args:
            reference_model_inputs ([`Mapping[str, Tensor]`):
                Reference inputs for the model.

        Returns:
            `Mapping[str, Tensor]`: The mapping holding the kwargs to provide to the model's forward function
        """
        return reference_model_inputs

    def patch_ops(self):
        for spec in self._patching_specs:
            custom_op = spec.custom_op if spec.op_wrapper is None else spec.op_wrapper(spec.custom_op)
            setattr(spec.o, spec.name, custom_op)

    def restore_ops(self):
        for spec in self._patching_specs:
            orig_op = spec.orig_op if spec.op_wrapper is None else spec.op_wrapper(spec.orig_op)
            setattr(spec.o, spec.name, orig_op)

    @classmethod
    def flatten_output_collection_property(cls, name: str, field: Iterable[Any]) -> Dict[str, Any]:
        """
        Flatten any potential nested structure expanding the name of the field with the index of the element within the
        structure.

        Args:
            name: The name of the nested structure
            field: The structure to, potentially, be flattened

        Returns:
            (Dict[str, Any]): Outputs with flattened structure and key mapping this new structure.

        """
        from itertools import chain



SUMMARY:
This code snippet is part of the Hugging Face Transformers library, specifically implementing some methods related to model patching and dummy data generation for models.

Here's a brief overview:

**Model Patching**

* The `patch_ops` method iterates over a list of `_patching_specs`, each of which contains information about a custom operation (`custom_op`) or wrapping it with another operator (`op_wrapper`).
* For each spec, it sets the original operator (`orig_op`) to be passed back after patching, effectively "restoring" its original behavior.

**Dummy Data Generation**

* The `generate_dummy_inputs`, `generate_dummy_inputs_onnxruntime`, and `generate_dummy_images` methods generate dummy data for models based on their input names.
	+ For tokenizers, it creates a list of sequences with the same sequence length as the model, using the special token (`input_token`) to pad the inputs.
	+ For image processing models, it generates images with the specified dimensions and number of channels (num_channels), padding the height and width if necessary.
	+ For feature extraction models, it generates audio data with the specified sampling rate, duration, and frequency.

**Utility Function**

* The `flatten_output_collection_property` method takes a nested structure (`field`) as input, flattens it by appending indices to each key, and returns the resulting flattened dictionary. 

This code snippet seems to be part of a larger library or framework for building and deploying machine learning models, specifically designed for text processing tasks. It provides methods for generating dummy data that can be used for testing, validation, and inference purposes.


key: chroma:document
string_value:
CODE:

class OnnxConfigWithPast(OnnxConfig, ABC):
    def __init__(
        self,
        config: "PretrainedConfig",
        task: str = "default",
        patching_specs: List[PatchingSpec] = None,
        use_past: bool = False,
    ):
        super().__init__(config, task=task, patching_specs=patching_specs)
        self.use_past = use_past

    @classmethod
    def with_past(cls, config: "PretrainedConfig", task: str = "default") -> "OnnxConfigWithPast":
        """
        Instantiate a OnnxConfig with `use_past` attribute set to True

        Args:
            config: The underlying model's config to use when exporting to ONNX

        Returns:
            OnnxConfig with `.use_past = True`
        """
        return cls(config, task=task, use_past=True)

    @property
    def outputs(self) -> Mapping[str, Mapping[int, str]]:
        common_outputs = super().outputs
        if self.use_past:
            self.fill_with_past_key_values_(common_outputs, direction="outputs")

        return common_outputs

    @property
    def values_override(self) -> Optional[Mapping[str, Any]]:
        if hasattr(self._config, "use_cache"):
            return {"use_cache": self.use_past}

        return None

    @property
    def num_layers(self) -> int:
        """
        The number of layers attribute retrieved from the model config. Override this for model configs where the
        number of layers attribute is not called `num_layers`.
        """
        if not hasattr(self._config, "num_layers"):
            raise AttributeError(
                "could not find the number of layers attribute in the model configuration, override the num_layers"
                " property of the model OnnxConfig to solve this"
            )
        return self._config.num_layers

    @property
    def num_attention_heads(self) -> int:
        """
        The number of attention heads attribute retrieved from the model config. Override this for model configs where
        the number of attention heads attribute is not called `num_attention_heads`.
        """
        if not hasattr(self._config, "num_attention_heads"):
            raise AttributeError(
                "could not find the number of attention heads attribute in the model configuration, override the"
                " num_attention_heads property of the model OnnxConfig to solve this"
            )
        return self._config.num_attention_heads

    def generate_dummy_inputs(
        self,
        tokenizer: "PreTrainedTokenizerBase",
        batch_size: int = -1,
        seq_length: int = -1,
        is_pair: bool = False,
        framework: Optional[TensorType] = None,
    ) -> Mapping[str, Any]:
        # TODO: should we set seq_length = 1 when self.use_past = True?
        common_inputs = super().generate_dummy_inputs(
            tokenizer, batch_size=batch_size, seq_length=seq_length, is_pair=is_pair, framework=framework
        )

        if self.use_past:
            if not is_torch_available():
                raise ValueError("Cannot generate dummy past_keys inputs without PyTorch installed.")
            else:
                import torch

            batch, seqlen = common_inputs["input_ids"].shape
            # Not using the same length for past_key_values
            past_key_values_length = seqlen + 2
            shape = (
                batch,
                self.num_attention_heads,
                past_key_values_length,
                self._config.hidden_size // self.num_attention_heads,
            )

            if "attention_mask" in common_inputs:
                mask_dtype = common_inputs["attention_mask"].dtype
                common_inputs["attention_mask"] = torch.cat(
                    [common_inputs["attention_mask"], torch.ones(batch, past_key_values_length, dtype=mask_dtype)],
                    dim=1,
                )

            common_inputs["past_key_values"] = []
            for _ in range(self.num_layers):
                common_inputs["past_key_values"].append((torch.zeros(shape), torch.zeros(shape)))

        return common_inputs

    def fill_with_past_key_values_(
        self, inputs_or_outputs: Mapping[str, Mapping[int, str]], direction: str, inverted_values_shape: bool = False
    ):
        """
        Fill the input_or_outputs mapping with past_key_values dynamic axes considering.

        Args:
            inputs_or_outputs: The mapping to fill.
            direction: either "inputs" or "outputs", it specifies whether input_or_outputs is the input mapping or the
                output mapping, this is important for axes naming.
            inverted_values_shape:
                If `True`, store values on dynamic axis 1, else on axis 2.

        """
        if direction not in ["inputs", "outputs"]:
            raise ValueError(f'direction must either be "inputs" or "outputs", but {direction} was given')

        name = "past_key_values" if direction == "inputs" else "present"
        for i in range(self.num_layers):
            inputs_or_outputs[f"{name}.{i}.key"] = {0: "batch", 2: "past_sequence + sequence"}
            if inverted_values_shape:
                inputs_or_outputs[f"{name}.{i}.value"] = {0: "batch", 1: "past_sequence + sequence"}
            else:
                inputs_or_outputs[f"{name}.{i}.value"] = {0: "batch", 2: "past_sequence + sequence"}

    def _flatten_past_key_values_(self, flattened_output, name, idx, t):
        flattened_output[f"{name}.{idx}.key"] = t[0]
        flattened_output[f"{name}.{idx}.value"] = t[1]

    def flatten_output_collection_property(self, name: str, field: Iterable[Any]) -> Dict[str, Any]:
        flattened_output = {}
        if name in ["present", "past_key_values"]:
            for idx, t in enumerate(field):
                self._flatten_past_key_values_(flattened_output, name, idx, t)
        else:
            flattened_output = super().flatten_output_collection_property(name, field)



SUMMARY:
**Class Summary: OnnxConfigWithPast**

 inherits from `OnnxConfig` (ABC)

**Purpose:** The primary purpose of this class is to extend the functionality of the basic `OnnxConfig` by enabling the use of past key values in Onnx models.

**Key Implementation Details:**

*   Override properties to access and manipulate model configuration attributes (`num_layers`, `num_attention_heads`, and others).
*   Provide a method for generating dummy inputs with past key values (`generate_dummy_inputs`).

```
from typing import List, Mapping, Any, Optional

class OnnxConfigWithPast(OnnxConfig, ABC):
    def __init__(
        self,
        config: "PretrainedConfig",
        task: str = "default",
        patching_specs: List[PatchingSpec] = None,
        use_past: bool = False,
    ):
        super().__init__(config, task=task, patching_specs=patching_specs)
        self.use_past = use_past

    @classmethod
    def with_past(cls, config: "PretrainedConfig", task: str = "default") -> "OnnxConfigWithPast":
        # Instantiate OnnxConfig with `.use_past` attribute set to True.
```

Notable Dependencies:

*   `pretrainedconfig` module for model configuration
*   `PatchingSpec` and `TensorType` modules (not shown in code snippet)

**Interfaces Implemented:**

*   `OnnxConfig`
*   `ABC` (Abstract Base Class)
```


key: chroma:document
string_value:
CODE:

class OnnxSeq2SeqConfigWithPast(OnnxConfigWithPast):
    @property
    def outputs(self) -> Mapping[str, Mapping[int, str]]:
        common_outputs = super(OnnxConfigWithPast, self).outputs
        # Renaming the outputs axes properly.
        for name, axes_names in common_outputs.items():
            sequence_name = "encoder_sequence" if "encoder" in name else "decoder_sequence"
            for axis_idx, name in axes_names.items():
                if "sequence" in name:
                    axes_names[axis_idx] = sequence_name
                # We reset the value as the order in common_outputs (OrderedDict) is lost otherwise
                else:
                    axes_names[axis_idx] = name
        if self.use_past:
            self.fill_with_past_key_values_(common_outputs, direction="outputs")

        return common_outputs

    @property
    def num_layers(self) -> Tuple[int]:
        try:
            num_layers = super().num_layers
            num_layers = (num_layers, num_layers)
        except AttributeError:
            if hasattr(self._config, "encoder_layers") and hasattr(self._config, "decoder_layers"):
                num_layers = (self._config.encoder_layers, self._config.decoder_layers)
            else:
                raise AttributeError(
                    "could not find the number of encoder and decoder layers attributes in the model configuration,"
                    " override the num_layers property of the model OnnxConfig to solve this"
                )

        return num_layers

    @property
    def num_attention_heads(self) -> Tuple[int]:
        try:
            num_attention_heads = super().num_attention_heads
            num_attention_heads = (num_attention_heads, num_attention_heads)
        except AttributeError:
            if hasattr(self._config, "encoder_attention_heads") and hasattr(self._config, "decoder_attention_heads"):
                num_attention_heads = (self._config.encoder_attention_heads, self._config.decoder_attention_heads)
            else:
                raise AttributeError(
                    "could not find the number of attention heads for the encoder and the decoder attributes in the"
                    " model configuration, override the num_attention_heads property of the model OnnxConfig to solve"
                    " this"
                )
        return num_attention_heads

    def generate_dummy_inputs(
        self,
        tokenizer: "PreTrainedTokenizerBase",
        batch_size: int = -1,
        seq_length: int = -1,
        is_pair: bool = False,
        framework: Optional[TensorType] = None,
    ) -> Mapping[str, Any]:
        encoder_inputs = super(OnnxConfigWithPast, self).generate_dummy_inputs(
            tokenizer, batch_size=batch_size, seq_length=seq_length, is_pair=is_pair, framework=framework
        )

        # Generate decoder inputs
        decoder_seq_length = seq_length if not self.use_past else 1
        decoder_inputs = super(OnnxConfigWithPast, self).generate_dummy_inputs(
            tokenizer, batch_size=batch_size, seq_length=decoder_seq_length, is_pair=is_pair, framework=framework
        )
        decoder_inputs = {f"decoder_{name}": tensor for name, tensor in decoder_inputs.items()}
        common_inputs = dict(**encoder_inputs, **decoder_inputs)

        if self.use_past:
            if not is_torch_available():
                raise ValueError("Cannot generate dummy past_keys inputs without PyTorch installed.")
            else:
                import torch
            batch = common_inputs["input_ids"].shape[0]
            encoder_seq_length = common_inputs["input_ids"].shape[1]
            decoder_seq_length = common_inputs["decoder_input_ids"].shape[1]
            num_encoder_attention_heads, num_decoder_attention_heads = self.num_attention_heads
            encoder_shape = (
                batch,
                num_encoder_attention_heads,
                encoder_seq_length,
                self._config.hidden_size // num_encoder_attention_heads,
            )
            decoder_shape = (
                batch,
                num_decoder_attention_heads,
                # Not using the same length for past_key_values
                decoder_seq_length + 3,
                self._config.hidden_size // num_decoder_attention_heads,
            )

            common_inputs["past_key_values"] = []
            # If the number of encoder and decoder layers are present in the model configuration, both are considered
            num_encoder_layers, num_decoder_layers = self.num_layers
            min_num_layers = min(num_encoder_layers, num_decoder_layers)
            max_num_layers = max(num_encoder_layers, num_decoder_layers) - min_num_layers
            remaining_side_name = "encoder" if num_encoder_layers > num_decoder_layers else "decoder"

            for _ in range(min_num_layers):
                # For encoder-decoder models, past_key_values contains pre-computed values for both the encoder and the
                # decoder layers, hence a tuple of 4 tensors instead of 2
                common_inputs["past_key_values"].append(
                    (
                        torch.zeros(decoder_shape),
                        torch.zeros(decoder_shape),
                        torch.zeros(encoder_shape),
                        torch.zeros(encoder_shape),
                    )
                )

            # TODO: test this.
            shape = encoder_shape if remaining_side_name == "encoder" else decoder_shape
            for _ in range(min_num_layers, max_num_layers):
                common_inputs["past_key_values"].append((torch.zeros(shape), torch.zeros(shape)))

        return common_inputs

    def fill_with_past_key_values_(self, inputs_or_outputs: Mapping[str, Mapping[int, str]], direction: str):
        if direction not in ["inputs", "outputs"]:
            raise ValueError(f'direction must either be "inputs" or "outputs", but {direction} was given')

        name = "past_key_values" if direction == "inputs" else "present"

        # If the number of encoder and decoder layers are present in the model configuration, both are considered
        num_encoder_layers, num_decoder_layers = self.num_layers
        min_num_layers = min(num_encoder_layers, num_decoder_layers)
        max_num_layers = max(num_encoder_layers, num_decoder_layers) - min_num_layers
        remaining_side_name = "encoder" if num_encoder_layers > num_decoder_layers else "decoder"

        encoder_sequence = "past_encoder_sequence"
        decoder_sequence = "past_decoder_sequence" if direction == "inputs" else "past_decoder_sequence + sequence"

        for i in range(min_num_layers):
            inputs_or_outputs[f"{name}.{i}.decoder.key"] = {0: "batch", 2: decoder_sequence}
            inputs_or_outputs[f"{name}.{i}.decoder.value"] = {0: "batch", 2: decoder_sequence}
            inputs_or_outputs[f"{name}.{i}.encoder.key"] = {0: "batch", 2: encoder_sequence}
            inputs_or_outputs[f"{name}.{i}.encoder.value"] = {0: "batch", 2: encoder_sequence}

        for i in range(min_num_layers, max_num_layers):
            if remaining_side_name == "encoder":
                axes_info = {0: "batch", 2: encoder_sequence}
            else:
                axes_info = {0: "batch", 2: decoder_sequence}
            inputs_or_outputs[f"{name}.{i}.{remaining_side_name}.key"] = axes_info

    def _flatten_past_key_values_(self, flattened_output, name, idx, t):
        flattened_output[f"{name}.{idx}.decoder.key"] = t[0]
        flattened_output[f"{name}.{idx}.decoder.value"] = t[1]
        flattened_output[f"{name}.{idx}.encoder.key"] = t[2]


SUMMARY:
**Class Summary: OnnxSeq2SeqConfigWithPast**

This class represents the configuration for a sequence-to-sequence model using the ONNX format, with support for past key values.

**Main Purpose and Functionality**
The main purpose of this class is to extend the `OnnxConfigWithPast` base class with additional features for sequence-to-sequence models. It provides properties for accessing output axes, layer counts, attention head counts, and generates dummy input tensors for encoding and decoding sequences.

**Key Implementation Details**

*   **Renamed Output Axes**: The `outputs` property renames the output axes to follow a consistent naming convention.
*   **num_layers**, **num_attention_heads**: These properties return tuples of encoder and decoder layer counts and attention head counts, respectively.
*   **generate_dummy_inputs**: This method generates dummy input tensors for encoding and decoding sequences.

**Notable Dependencies or Requirements**

*   `OnnxConfigWithPast`: The base class used to extend with additional features.
*   `PreTrainedTokenizerBase`: Tokenizer required for generating dummy inputs.
*   `torch`: Python tensor library used for creating input tensors.


key: chroma:document
string_value:
CODE:

class ReturnType(enum.Enum):
    TENSORS = 0
    NEW_TEXT = 1


SUMMARY:
**Summary:**

The `ReturnType` enum class is defined to represent different types of return values. It has three variants:

- `TENSORS`: returns tensor data (value: 0)
- `NEW_TEXT`: returns new text data (value: 1)
- `FULL_TEXT`: returns full text data (value: 2)

**Key Implementation Details:**

The class uses the `enum` module to define an enumerative type. This is a good design choice for consistency and readability.

Notable Dependency:
This enum has no explicit dependencies, but it might be used within another module that imports this enum.


key: chroma:document
string_value:
CODE:

class Chat:
    """This class is intended to just be used internally in this pipeline and not exposed to users. We convert chats
    to this format because the rest of the pipeline code tends to assume that lists of messages are
    actually a batch of samples rather than messages in the same conversation."""

    def __init__(self, messages: Dict):
        for message in messages:
            if not ("role" in message and "content" in message):
                raise ValueError("When passing chat dicts as input, each dict must have a 'role' and 'content' key.")


SUMMARY:
**Class Summary**

The `Chat` class represents a collection of messages from a conversation. It validates incoming message dictionaries to ensure they contain both `role` and `content` keys, then stores the messages in an instance variable.

**Key Implementation Details**

*   **Validation**: The `__init__` method checks each message dictionary for required keys (`role` and `content`) and raises a `ValueError` if invalid.
*   **Message Storage**: Messages are stored in an instance variable `self.messages`, which is a dictionary containing the collected messages.

**Notable Dependencies**

No external dependencies.


key: chroma:document
string_value:
CODE:

def ffmpeg_read(bpayload: bytes, sampling_rate: int) -> np.array:
    """
    Helper function to read an audio file through ffmpeg.
    """
    ar = f"{sampling_rate}"
    ac = "1"
    format_for_conversion = "f32le"
    ffmpeg_command = [
        "ffmpeg",
        "-i",
        "pipe:0",
        "-ac",
        ac,
        "-ar",
        ar,
        "-f",
        format_for_conversion,
        "-hide_banner",
        "-loglevel",
        "quiet",
        "pipe:1",
    ]

    try:
        ffmpeg_process = subprocess.Popen(ffmpeg_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    except FileNotFoundError:
        raise ValueError("ffmpeg was not found but is required to load audio files from filename")
    output_stream = ffmpeg_process.communicate(bpayload)
    out_bytes = output_stream[0]

    audio = np.frombuffer(out_bytes, np.float32)
    if audio.shape[0] == 0:
        raise ValueError("Malformed soundfile")


SUMMARY:
**Code Summary**

### Overview

*   Reads an audio file using FFmpeg for conversion to NumPy arrays.
*   Parameters: `bpayload` (bytes input), `sampling_rate` (audio sampling rate).

### Function Signature
```python
def ffmpeg_read(bpayload: bytes, sampling_rate: int) -> np.array:
```

### Primary Purpose

Reads audio data from a file using FFmpeg, converting it to a NumPy array.

### Key Implementation Details

*   Uses `subprocess` to run an FFmpeg command with specified input and output streams.
*   Converting output to a NumPy array of floating-point values (`f32le` format).
*   Handles potential `ValueError` exceptions for malformed audio files or missing FFmpeg installation.

### Notable Dependencies

*   External dependency: `ffmpeg` (FFmpeg executable) must be installed on the system.
*   Related classes/functions:
    *   `subprocess`
    *   `np.array`, `np.float32`

### Return Value
```python
-> np.array
```

The function returns a NumPy array representing the audio data.


key: chroma:document
string_value:
CODE:

class ReturnType(enum.Enum):
    TENSORS = 0


SUMMARY:
**Summary:**

This `ReturnType` enum defines a set of named values used to specify the output type. It has two values:

* `TENSORS`: represents tensor data (value: 0)
* `TEXT`: represents text data (value: 1)

Notable details:

* Uses Python's built-in `enum.Enum` class.
* No external dependencies are listed, assuming it is self-contained within the module.


key: chroma:document
string_value:
CODE:

class TokenClassificationArgumentHandler(ArgumentHandler):
    """
    Handles arguments for token classification.
    """

    def __call__(self, inputs: Union[str, List[str]], **kwargs):
        if inputs is not None and isinstance(inputs, (list, tuple)) and len(inputs) > 0:
            inputs = list(inputs)
            batch_size = len(inputs)
        elif isinstance(inputs, str):
            inputs = [inputs]
            batch_size = 1
        elif Dataset is not None and isinstance(inputs, Dataset) or isinstance(inputs, types.GeneratorType):
            return inputs, None
        else:
            raise ValueError("At least one input is required.")

        offset_mapping = kwargs.get("offset_mapping")
        if offset_mapping:
            if isinstance(offset_mapping, list) and isinstance(offset_mapping[0], tuple):
                offset_mapping = [offset_mapping]
            if len(offset_mapping) != batch_size:
                raise ValueError("offset_mapping should have the same batch size as the input")


SUMMARY:
**TokenClassificationArgumentHandler Class Summary**

### Purpose and Functionality
The `TokenClassificationArgumentHandler` class is responsible for handling arguments for token classification tasks. It takes an input (string or list of strings) and provides the necessary configuration (`offset_mapping`) to process the data.

### Key Implementation Details

* **Input Processing**: The input is converted to a list, and its length is used to determine the batch size.
* **Configuration Validation**: The `offset_mapping` parameter is validated to ensure it matches the batch size of the input.

### Notable Dependencies and Requirements

* `ArgumentHandler`: Parent class
* `Dataset`: Optional parent class or generator type
* `types.GeneratorType`: Conditional check type
* `kwargs`: Configurations, including `offset_mapping`

**Notes:**

* The class returns two values: input data (`inputs`) and `offset_mapping` configuration.
* Validation checks are in place to ensure proper handling of input types.


key: chroma:document
string_value:
CODE:

class AggregationStrategy(ExplicitEnum):
    """All the valid aggregation strategies for TokenClassificationPipeline"""

    NONE = "none"
    SIMPLE = "simple"
    FIRST = "first"
    AVERAGE = "average"


SUMMARY:
**Summary:**

The `AggregationStrategy` class represents an enumeration of valid aggregation strategies for a `TokenClassificationPipeline`. It defines five distinct strategies:
	- `NONE`
	- `SIMPLE`
	- `FIRST`
	- `AVERAGE`
	- `MAX`

**Implementation Details:**

No key implementation details are provided; this is an explicit enumeration class.

**Notable Dependencies/Requirements:**

* No external dependencies or requirements listed.
* Interfaces: Implied by the class's use in a `TokenClassificationPipeline`.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/src/transformers/pipelines/automatic_speech_recognition.py


key: chroma:document
string_value:
CODE:

def rescale_stride(stride, ratio):
    """
    Rescales the stride values from audio space to tokens/logits space.

    (160_000, 16_000, 16_000) -> (2000, 200, 200) for instance.
    """
    # Shape is [B, SEQ] for tokens
    # [B, SEQ, V] for logits

    new_strides = []
    for input_n, left, right in stride:
        token_n = int(round(input_n * ratio))
        left = int(round(left / input_n * token_n))
        right = int(round(right / input_n * token_n))
        new_stride = (token_n, left, right)
        new_strides.append(new_stride)



SUMMARY:
**Function Summary: `rescale_stride`**

**Purpose and Functionality**
Rescales stride values from audio space to tokens/logits space.

**Key Implementation Details**

* Calculates new stride dimensions based on input stride dimensions and scaling ratio (`ratio`)
* Uses integer arithmetic to round intermediate results
* Appends transformed strides to a list (`new_strides`)

**Notable Dependencies/Requirements**

* Input `stride`: expected shape `[B, SEQ]`, or `[B, SEQ, V]`
* Scaling ratio: `ratio` (no specific type or range indicated)
* External dependency: `int.round()` and arithmetic operations


key: chroma:document
string_value:
CODE:

def chunk_iter(inputs, feature_extractor, chunk_len, stride_left, stride_right, dtype=None):
    inputs_len = inputs.shape[0]
    step = chunk_len - stride_left - stride_right
    for chunk_start_idx in range(0, inputs_len, step):
        chunk_end_idx = chunk_start_idx + chunk_len
        chunk = inputs[chunk_start_idx:chunk_end_idx]
        processed = feature_extractor(chunk, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt")
        if dtype is not None:
            processed = processed.to(dtype=dtype)
        _stride_left = 0 if chunk_start_idx == 0 else stride_left
        is_last = chunk_end_idx >= inputs_len
        _stride_right = 0 if is_last else stride_right

        chunk_len = chunk.shape[0]
        stride = (chunk_len, _stride_left, _stride_right)
        if chunk.shape[0] > _stride_left:
            yield {"is_last": is_last, "stride": stride, **processed}
        if is_last:


SUMMARY:
**Function Summary:**

`chunk_iter`: An iterator function that processes input data in chunks based on user-defined parameters. It leverages a `feature_extractor` to extract features from each chunk and yields intermediate results.

**Key Details:**

* Inputs are processed in chunks of length `chunk_len`, with striding applied by `stride_left` and `stride_right`.
* The function extracts audio features using `feature_extractor` and returns tensors.
* It maintains an optional output data type `dtype`.
* Yields dictionaries containing feature values, along with stride information and processed tensors.

**Notable Dependencies/Requirements:**

* `feature_extractor`
* `sampling_rate` attribute (from `feature_extractor`)
* Output tensor shape and data type (`processed`)


key: chroma:document
string_value:
CODE:

def _fast_find_longest_common_sequence(sequence_left, sequence_right):
    seq_len_left = len(sequence_left)
    seq_len_right = len(sequence_right)
    counter = [[0] * (seq_len_right + 1) for _ in range(seq_len_left + 1)]
    longest = 0
    for i in range(seq_len_left):
        for j in range(seq_len_right):
            if sequence_left[i] == sequence_right[j]:
                previous_counter = counter[i][j] + 1
                counter[i + 1][j + 1] = previous_counter
                if previous_counter > longest:
                    longest = previous_counter

    counter = np.array(counter)
    # we return the idx of the first element of the longest common sequence in the left sequence
    index_left = np.argwhere(counter == longest)[-1][0] - longest if longest != 0 else -1
    index_right = np.argwhere(counter == longest)[-1][1] - longest if longest != 0 else -1


SUMMARY:
**Function Summary: `_fast_find_longest_common_sequence`**

This function finds the longest common subsequence (LCS) between two input sequences and returns its starting indices and length in both sequences.

**Purposes and Functionality:**

* Calculates the LCS using dynamic programming with a nested counter matrix.
* Compares characters at corresponding positions in both sequences to build the LCS.
* Returns:
	+ `index_left`: Index of the first element of the LCS in the `sequence_left`.
	+ `index_right`: Index of the first element of the LCS in the `sequence_right`.
	+ `longest`: Length of the longest common subsequence.

**Key Implementation Details:**

* Uses a 2D counter matrix (`counter`) to store intermediate results.
* Failsafe checks to handle cases where the input sequences are empty or have no matching characters.
* Utilizes NumPy's `argwhere` function to find the indices of the maximum value in the counter matrix.

**Notable Dependencies and Requirements:**

* Python 3.x compatibility
* NumPy library for efficient array operations
* No external dependencies on third-party libraries or APIs.


key: chroma:document
string_value:
CODE:

def _find_longest_common_sequence(sequences, tokenizer):
    # TODO  Use a faster algorithm this can probably be done in O(n)
    # using suffix array.
    # It might be tedious to do because of fault tolerance.
    # We actually have a really good property which is that the total sequence
    # MUST be those subsequences in order.
    # Also the algorithm should be more tolerant to errors.
    sequence = [tok_id for tok_id in sequences[0][0].tolist() if tok_id not in tokenizer.all_special_ids]
    for new_seq in sequences[1:]:
        new_sequence = [tok_id for tok_id in new_seq[0].tolist() if tok_id not in tokenizer.all_special_ids]

        index = 0
        max_ = 0.0
        for i in range(1, len(new_sequence) + 1):
            # epsilon to favor long perfect matches
            eps = i / 10000.0
            matches = np.sum(np.array(sequence[-i:]) == np.array(new_sequence[:i]))
            matching = matches / i + eps
            if matches > 1 and matching > max_:
                index = i
                max_ = matching
        sequence.extend(new_sequence[index:])


SUMMARY:
**Summary:**

### `_find_longest_common_sequence` Function 

#### Main Purpose & Functionality:

This function finds the longest common subsequence (LCS) among input sequences. It filters out special IDs from tokenized sequences and uses dynamic programming with epsilon optimization to measure match tolerance.

#### Key Implementation Details:

* **Sequence Filtering:** Removes special IDs from each sequence.
* **Dynamic Programming w/ Epsilon Optimizations:**
	+ Measures matches using Euclidean distance (`np.sum(np.array(sequence[-i:]) == np.array(new_sequence[:i]))`)
	+ Introduces epsilon for favorable long perfect matches (`matches / i + eps`)

#### Notable Dependencies & Requirements:

* `tokenizer`: instance with `all_special_ids`
* `numpy` (alias: `np`) for numerical computations

#### Method Signatures & Interfaces:

* None specific, but operates on sequences treated as numpy arrays.
* Extends sequence through LCP indexing.


key: chroma:document
string_value:
CODE:

class AutomaticSpeechRecognitionPipeline(ChunkPipeline):
    """
    Pipeline that aims at extracting spoken text contained within some audio.

    The input can be either a raw waveform or a audio file. In case of the audio file, ffmpeg should be installed for
    to support multiple audio formats

    Example:

    ```python
    >>> from transformers import pipeline

    >>> transcriber = pipeline(model="openai/whisper-base")
    >>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac")
    {'text': ' He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered flour-fatten sauce.'}
    ```

    Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)

    Arguments:
        model ([`PreTrainedModel`] or [`TFPreTrainedModel`]):
            The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from
            [`PreTrainedModel`] for PyTorch and [`TFPreTrainedModel`] for TensorFlow.
        feature_extractor ([`SequenceFeatureExtractor`]):
            The feature extractor that will be used by the pipeline to encode waveform for the model.
        tokenizer ([`PreTrainedTokenizer`]):
            The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from
            [`PreTrainedTokenizer`].
        decoder (`pyctcdecode.BeamSearchDecoderCTC`, *optional*):
            [PyCTCDecode's
            BeamSearchDecoderCTC](https://github.com/kensho-technologies/pyctcdecode/blob/2fd33dc37c4111417e08d89ccd23d28e9b308d19/pyctcdecode/decoder.py#L180)
            can be passed for language model boosted decoding. See [`Wav2Vec2ProcessorWithLM`] for more information.
        chunk_length_s (`float`, *optional*, defaults to 0):
            The input length for in each chunk. If `chunk_length_s = 0` then chunking is disabled (default).

            <Tip>

            For more information on how to effectively use `chunk_length_s`, please have a look at the [ASR chunking
            blog post](https://huggingface.co/blog/asr-chunking).

            </Tip>

        stride_length_s (`float`, *optional*, defaults to `chunk_length_s / 6`):
            The length of stride on the left and right of each chunk. Used only with `chunk_length_s > 0`. This enables
            the model to *see* more context and infer letters better than without this context but the pipeline
            discards the stride bits at the end to make the final reconstitution as perfect as possible.

            <Tip>

            For more information on how to effectively use `stride_length_s`, please have a look at the [ASR chunking
            blog post](https://huggingface.co/blog/asr-chunking).

            </Tip>

        framework (`str`, *optional*):
            The framework to use, either `"pt"` for PyTorch or `"tf"` for TensorFlow. The specified framework must be
            installed. If no framework is specified, will default to the one currently installed. If no framework is
            specified and both frameworks are installed, will default to the framework of the `model`, or to PyTorch if
            no model is provided.
        device (Union[`int`, `torch.device`], *optional*):
            Device ordinal for CPU/GPU supports. Setting this to `None` will leverage CPU, a positive will run the
            model on the associated CUDA device id.
        torch_dtype (Union[`int`, `torch.dtype`], *optional*):
            The data-type (dtype) of the computation. Setting this to `None` will use float32 precision. Set to
            `torch.float16` or `torch.bfloat16` to use half-precision in the respective dtypes.

    """

    def __init__(
        self,
        model: "PreTrainedModel",
        feature_extractor: Union["SequenceFeatureExtractor", str] = None,
        tokenizer: Optional[PreTrainedTokenizer] = None,
        decoder: Optional[Union["BeamSearchDecoderCTC", str]] = None,
        device: Union[int, "torch.device"] = None,
        torch_dtype: Optional[Union[str, "torch.dtype"]] = None,
        **kwargs,
    ):
        # set the model type so we can check we have the right pre- and post-processing parameters
        if model.config.model_type == "whisper":
            self.type = "seq2seq_whisper"
        elif model.__class__.__name__ in MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMES.values():
            self.type = "seq2seq"
        elif (
            feature_extractor._processor_class
            and feature_extractor._processor_class.endswith("WithLM")
            and decoder is not None
        ):
            self.decoder = decoder
            self.type = "ctc_with_lm"
        else:
            self.type = "ctc"

        super().__init__(model, tokenizer, feature_extractor, device=device, torch_dtype=torch_dtype, **kwargs)

    def __call__(
        self,
        inputs: Union[np.ndarray, bytes, str],
        **kwargs,
    ):
        """
        Transcribe the audio sequence(s) given as inputs to text. See the [`AutomaticSpeechRecognitionPipeline`]
        documentation for more information.

        Args:
            inputs (`np.ndarray` or `bytes` or `str` or `dict`):
                The inputs is either :
                    - `str` that is either the filename of a local audio file, or a public URL address to download the
                      audio file. The file will be read at the correct sampling rate to get the waveform using
                      *ffmpeg*. This requires *ffmpeg* to be installed on the system.
                    - `bytes` it is supposed to be the content of an audio file and is interpreted by *ffmpeg* in the
                      same way.
                    - (`np.ndarray` of shape (n, ) of type `np.float32` or `np.float64`)
                        Raw audio at the correct sampling rate (no further check will be done)
                    - `dict` form can be used to pass raw audio sampled at arbitrary `sampling_rate` and let this
                      pipeline do the resampling. The dict must be in the format `{"sampling_rate": int, "raw":
                      np.array}` with optionally a `"stride": (left: int, right: int)` than can ask the pipeline to
                      treat the first `left` samples and last `right` samples to be ignored in decoding (but used at
                      inference to provide more context to the model). Only use `stride` with CTC models.
            return_timestamps (*optional*, `str` or `bool`):
                Only available for pure CTC models (Wav2Vec2, HuBERT, etc) and the Whisper model. Not available for
                other sequence-to-sequence models.

                For CTC models, timestamps can take one of two formats:
                    - `"char"`: the pipeline will return timestamps along the text for every character in the text. For
                        instance, if you get `[{"text": "h", "timestamp": (0.5, 0.6)}, {"text": "i", "timestamp": (0.7,
                        0.9)}]`, then it means the model predicts that the letter "h" was spoken after `0.5` and before
                        `0.6` seconds.
                    - `"word"`: the pipeline will return timestamps along the text for every word in the text. For
                        instance, if you get `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text": "there", "timestamp":
                        (1.0, 1.5)}]`, then it means the model predicts that the word "hi" was spoken after `0.5` and
                        before `0.9` seconds.

                For the Whisper model, timestamps can take one of two formats:
                    - `"word"`: same as above for word-level CTC timestamps. Word-level timestamps are predicted
                        through the *dynamic-time warping (DTW)* algorithm, an approximation to word-level timestamps
                        by inspecting the cross-attention weights.
                    - `True`: the pipeline will return timestamps along the text for *segments* of words in the text.
                        For instance, if you get `[{"text": " Hi there!", "timestamp": (0.5, 1.5)}]`, then it means the
                        model predicts that the segment "Hi there!" was spoken after `0.5` and before `1.5` seconds.
                        Note that a segment of text refers to a sequence of one or more words, rather than individual
                        words as with word-level timestamps.
            generate_kwargs (`dict`, *optional*):
                The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a
                complete overview of generate, check the [following
                guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation).

        Return:
            `Dict`: A dictionary with the following keys:
                - **text** (`str`): The recognized text.
                - **chunks** (*optional(, `List[Dict]`)
                    When using `return_timestamps`, the `chunks` will become a list containing all the various text
                    chunks identified by the model, *e.g.* `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text":
                    "there", "timestamp": (1.0, 1.5)}]`. The original full text can roughly be recovered by doing
                    `"".join(chunk["text"] for chunk in output["chunks"])`.
        """
        return super().__call__(inputs, **kwargs)

    def _sanitize_parameters(
        self,
        chunk_length_s=None,
        stride_length_s=None,
        ignore_warning=None,
        decoder_kwargs=None,
        return_timestamps=None,
        return_language=None,
        generate_kwargs=None,
        max_new_tokens=None,
    ):
        # No parameters on this pipeline right now
        preprocess_params = {}
        if chunk_length_s is not None:
            if self.type == "seq2seq" and not ignore_warning:
                logger.warning(
                    "Using `chunk_length_s` is very experimental with seq2seq models. The results will not necessarily"
                    " be entirely accurate and will have caveats. More information:"
                    " https://github.com/huggingface/transformers/pull/20104. Ignore this warning with pipeline(...,"
                    " ignore_warning=True)"
                )
            preprocess_params["chunk_length_s"] = chunk_length_s
        if stride_length_s is not None:
            preprocess_params["stride_length_s"] = stride_length_s

        forward_params = defaultdict(dict)
        if max_new_tokens is not None:
            warnings.warn(
                "`max_new_tokens` is deprecated and will be removed in version 4.49 of Transformers. To remove this warning, pass `max_new_tokens` as a key inside `generate_kwargs` instead.",
                FutureWarning,
            )
            forward_params["max_new_tokens"] = max_new_tokens
        if generate_kwargs is not None:
            if max_new_tokens is not None and "max_new_tokens" in generate_kwargs:
                raise ValueError(
                    "`max_new_tokens` is defined both as an argument and inside `generate_kwargs` argument, please use"
                    " only 1 version"
                )
            forward_params.update(generate_kwargs)

        postprocess_params = {}
        if decoder_kwargs is not None:
            postprocess_params["decoder_kwargs"] = decoder_kwargs
        if return_timestamps is not None:
            # Check whether we have a valid setting for return_timestamps and throw an error before we perform a forward pass
            if self.type == "seq2seq" and return_timestamps:
                raise ValueError("We cannot return_timestamps yet on non-CTC models apart from Whisper!")
            if self.type == "ctc_with_lm" and return_timestamps != "word":
                raise ValueError("CTC with LM can only predict word level timestamps, set `return_timestamps='word'`")
            if self.type == "ctc" and return_timestamps not in ["char", "word"]:
                raise ValueError(
                    "CTC can either predict character level timestamps, or word level timestamps. "
                    "Set `return_timestamps='char'` or `return_timestamps='word'` as required."
                )
            if self.type == "seq2seq_whisper" and return_timestamps == "char":
                raise ValueError(
                    "Whisper cannot return `char` timestamps, only word level or segment level timestamps. "
                    "Use `return_timestamps='word'` or `return_timestamps=True` respectively."
                )
            forward_params["return_timestamps"] = return_timestamps
            postprocess_params["return_timestamps"] = return_timestamps
        if return_language is not None:
            if self.type != "seq2seq_whisper":
                raise ValueError("Only Whisper can return language for now.")
            postprocess_params["return_language"] = return_language

        if self.assistant_model is not None:
            forward_params["assistant_model"] = self.assistant_model
        if self.assistant_tokenizer is not None:
            forward_params["tokenizer"] = self.tokenizer
            forward_params["assistant_tokenizer"] = self.assistant_tokenizer

        return preprocess_params, forward_params, postprocess_params

    def preprocess(self, inputs, chunk_length_s=0, stride_length_s=None):
        if isinstance(inputs, str):
            if inputs.startswith("http://") or inputs.startswith("https://"):
                # We need to actually check for a real protocol, otherwise it's impossible to use a local file
                # like http_huggingface_co.png
                inputs = requests.get(inputs).content
            else:
                with open(inputs, "rb") as f:
                    inputs = f.read()

        if isinstance(inputs, bytes):
            inputs = ffmpeg_read(inputs, self.feature_extractor.sampling_rate)

        stride = None
        extra = {}
        if isinstance(inputs, dict):
            stride = inputs.pop("stride", None)
            # Accepting `"array"` which is the key defined in `datasets` for
            # better integration
            if not ("sampling_rate" in inputs and ("raw" in inputs or "array" in inputs)):
                raise ValueError(
                    "When passing a dictionary to AutomaticSpeechRecognitionPipeline, the dict needs to contain a "
                    '"raw" key containing the numpy array representing the audio and a "sampling_rate" key, '
                    "containing the sampling_rate associated with that array"
                )

            _inputs = inputs.pop("raw", None)
            if _inputs is None:
                # Remove path which will not be used from `datasets`.
                inputs.pop("path", None)
                _inputs = inputs.pop("array", None)
            in_sampling_rate = inputs.pop("sampling_rate")
            extra = inputs
            inputs = _inputs
            if in_sampling_rate != self.feature_extractor.sampling_rate:
                if is_torchaudio_available():
                    from torchaudio import functional as F
                else:
                    raise ImportError(
                        "torchaudio is required to resample audio samples in AutomaticSpeechRecognitionPipeline. "
                        "The torchaudio package can be installed through: `pip install torchaudio`."
                    )

                inputs = F.resample(
                    torch.from_numpy(inputs), in_sampling_rate, self.feature_extractor.sampling_rate
                ).numpy()
                ratio = self.feature_extractor.sampling_rate / in_sampling_rate
            else:
                ratio = 1
            if stride is not None:
                if stride[0] + stride[1] > inputs.shape[0]:
                    raise ValueError("Stride is too large for input")

                # Stride needs to get the chunk length here, it's going to get
                # swallowed by the `feature_extractor` later, and then batching
                # can add extra data in the inputs, so we need to keep track
                # of the original length in the stride so we can cut properly.
                stride = (inputs.shape[0], int(round(stride[0] * ratio)), int(round(stride[1] * ratio)))
        if not isinstance(inputs, np.ndarray):
            raise TypeError(f"We expect a numpy ndarray as input, got `{type(inputs)}`")
        if len(inputs.shape) != 1:
            raise ValueError("We expect a single channel audio input for AutomaticSpeechRecognitionPipeline")

        if chunk_length_s:
            if stride_length_s is None:
                stride_length_s = chunk_length_s / 6

            if isinstance(stride_length_s, (int, float)):
                stride_length_s = [stride_length_s, stride_length_s]

            # XXX: Carefuly, this variable will not exist in `seq2seq` setting.
            # Currently chunking is not possible at this level for `seq2seq` so
            # it's ok.
            align_to = getattr(self.model.config, "inputs_to_logits_ratio", 1)
            chunk_len = int(round(chunk_length_s * self.feature_extractor.sampling_rate / align_to) * align_to)
            stride_left = int(round(stride_length_s[0] * self.feature_extractor.sampling_rate / align_to) * align_to)
            stride_right = int(round(stride_length_s[1] * self.feature_extractor.sampling_rate / align_to) * align_to)

            if chunk_len < stride_left + stride_right:
                raise ValueError("Chunk length must be superior to stride length")

            for item in chunk_iter(
                inputs, self.feature_extractor, chunk_len, stride_left, stride_right, self.torch_dtype
            ):
                yield {**item, **extra}
        else:
            if self.type == "seq2seq_whisper" and inputs.shape[0] > self.feature_extractor.n_samples:
                processed = self.feature_extractor(
                    inputs,
                    sampling_rate=self.feature_extractor.sampling_rate,
                    truncation=False,
                    padding="longest",
                    return_tensors="pt",
                    return_attention_mask=True,
                )
            else:
                if self.type == "seq2seq_whisper" and stride is None:
                    processed = self.feature_extractor(
                        inputs,
                        sampling_rate=self.feature_extractor.sampling_rate,
                        return_tensors="pt",
                        return_token_timestamps=True,
                        return_attention_mask=True,
                    )
                    extra["num_frames"] = processed.pop("num_frames")
                else:
                    processed = self.feature_extractor(
                        inputs,
                        sampling_rate=self.feature_extractor.sampling_rate,
                        return_tensors="pt",
                        return_attention_mask=True,
                    )
            if self.torch_dtype is not None:
                processed = processed.to(dtype=self.torch_dtype)
            if stride is not None:
                if self.type == "seq2seq":
                    raise ValueError("Stride is only usable with CTC models, try removing it !")

                processed["stride"] = stride
            yield {"is_last": True, **processed, **extra}

    def _forward(self, model_inputs, return_timestamps=False, **generate_kwargs):
        attention_mask = model_inputs.pop("attention_mask", None)
        stride = model_inputs.pop("stride", None)
        num_frames = model_inputs.pop("num_frames", None)
        is_last = model_inputs.pop("is_last")

        if stride is not None and num_frames is not None:
            raise ValueError("num_frames must be used only when stride is None")

        if self.type in {"seq2seq", "seq2seq_whisper"}:
            # Consume values so we can let extra information flow freely through
            # the pipeline (important for `partial` in microphone)
            if "input_features" in model_inputs:
                inputs = model_inputs.pop("input_features")
            elif "input_values" in model_inputs:
                inputs = model_inputs.pop("input_values")
            else:
                raise ValueError(
                    "Seq2Seq speech recognition model requires either a "
                    f"`input_features` or `input_values` key, but only has {model_inputs.keys()}"
                )

            # custom processing for Whisper timestamps and word-level timestamps
            if return_timestamps and self.type == "seq2seq_whisper":
                generate_kwargs["return_timestamps"] = return_timestamps
                if return_timestamps == "word":
                    generate_kwargs["return_token_timestamps"] = True
                    generate_kwargs["return_segments"] = True

                    if stride is not None:
                        if isinstance(stride, tuple):
                            generate_kwargs["num_frames"] = stride[0] // self.feature_extractor.hop_length
                        else:
                            generate_kwargs["num_frames"] = [s[0] // self.feature_extractor.hop_length for s in stride]
                    else:
                        generate_kwargs["num_frames"] = num_frames

            # User-defined `generation_config` passed to the pipeline call take precedence
            if "generation_config" not in generate_kwargs:
                generate_kwargs["generation_config"] = self.generation_config

            tokens = self.model.generate(
                inputs=inputs,
                attention_mask=attention_mask,
                **generate_kwargs,
            )
            # whisper longform generation stores timestamps in "segments"
            if return_timestamps == "word" and self.type == "seq2seq_whisper":
                if "segments" not in tokens:
                    out = {"tokens": tokens["sequences"], "token_timestamps": tokens["token_timestamps"]}
                else:
                    token_timestamps = [
                        torch.cat([segment["token_timestamps"] for segment in segment_list])
                        for segment_list in tokens["segments"]
                    ]
                    out = {"tokens": tokens["sequences"], "token_timestamps": token_timestamps}
            else:
                out = {"tokens": tokens}
            if self.type == "seq2seq_whisper":
                if stride is not None:
                    out["stride"] = stride

        else:
            inputs = {
                self.model.main_input_name: model_inputs.pop(self.model.main_input_name),
                "attention_mask": attention_mask,
            }
            outputs = self.model(**inputs)
            logits = outputs.logits

            if self.type == "ctc_with_lm":
                out = {"logits": logits}
            else:
                out = {"tokens": logits.argmax(dim=-1)}
            if stride is not None:
                # Send stride to `postprocess`.
                # it needs to be handled there where
                # the pieces are to be concatenated.
                ratio = 1 / self.model.config.inputs_to_logits_ratio
                if isinstance(stride, tuple):
                    out["stride"] = rescale_stride([stride], ratio)[0]
                else:
                    out["stride"] = rescale_stride(stride, ratio)
        # Leftover
        extra = model_inputs
        return {"is_last": is_last, **out, **extra}

    def postprocess(
        self, model_outputs, decoder_kwargs: Optional[Dict] = None, return_timestamps=None, return_language=None
    ):
        # Optional return types
        optional = {}

        final_items = []
        key = "logits" if self.type == "ctc_with_lm" else "tokens"
        stride = None
        for outputs in model_outputs:
            if self.framework == "pt" and outputs[key].dtype in (torch.bfloat16, torch.float16):
                items = outputs[key].to(torch.float32).numpy()
            else:
                items = outputs[key].numpy()
            stride = outputs.get("stride", None)
            if stride is not None and self.type in {"ctc", "ctc_with_lm"}:
                total_n, left, right = stride
                # Total_n might be < logits.shape[1]
                # because of padding, that's why
                # we need to reconstruct this information
                # This won't work with left padding (which doesn't exist right now)
                right_n = total_n - right
                items = items[:, left:right_n]
            final_items.append(items)

        if stride and self.type == "seq2seq":
            items = _find_longest_common_sequence(final_items, self.tokenizer)
        elif self.type == "seq2seq_whisper":
            time_precision = self.feature_extractor.chunk_length / self.model.config.max_source_positions
            # Send the chunking back to seconds, it's easier to handle in whisper
            sampling_rate = self.feature_extractor.sampling_rate
            for output in model_outputs:
                if "stride" in output:
                    chunk_len, stride_left, stride_right = output["stride"]
                    # Go back in seconds
                    chunk_len /= sampling_rate
                    stride_left /= sampling_rate
                    stride_right /= sampling_rate
                    output["stride"] = chunk_len, stride_left, stride_right

            text, optional = self.tokenizer._decode_asr(
                model_outputs,
                return_timestamps=return_timestamps,
                return_language=return_language,
                time_precision=time_precision,
            )
        else:
            items = np.concatenate(final_items, axis=1)
            items = items.squeeze(0)

        if self.type == "ctc_with_lm":
            if decoder_kwargs is None:
                decoder_kwargs = {}
            beams = self.decoder.decode_beams(items, **decoder_kwargs)
            text = beams[0][0]
            if return_timestamps:
                # Simply cast from pyctcdecode format to wav2vec2 format to leverage
                # pre-existing code later
                chunk_offset = beams[0][2]
                offsets = []
                for word, (start_offset, end_offset) in chunk_offset:
                    offsets.append({"word": word, "start_offset": start_offset, "end_offset": end_offset})
        elif self.type != "seq2seq_whisper":
            skip_special_tokens = self.type != "ctc"
            text = self.tokenizer.decode(items, skip_special_tokens=skip_special_tokens)
            if return_timestamps:
                offsets = self.tokenizer.decode(
                    items, skip_special_tokens=skip_special_tokens, output_char_offsets=True
                )["char_offsets"]
                if return_timestamps == "word":
                    offsets = self.tokenizer._get_word_offsets(offsets, self.tokenizer.replace_word_delimiter_char)

        if return_timestamps and self.type not in {"seq2seq", "seq2seq_whisper"}:
            chunks = []
            for item in offsets:
                start = item["start_offset"] * self.model.config.inputs_to_logits_ratio
                start /= self.feature_extractor.sampling_rate

                stop = item["end_offset"] * self.model.config.inputs_to_logits_ratio
                stop /= self.feature_extractor.sampling_rate

                chunks.append({"text": item[return_timestamps], "timestamp": (start, stop)})
            optional["chunks"] = chunks

        extra = defaultdict(list)
        for output in model_outputs:
            output.pop("tokens", None)
            output.pop("logits", None)
            output.pop("is_last", None)
            output.pop("stride", None)
            output.pop("token_timestamps", None)
            for k, v in output.items():
                extra[k].append(v)


SUMMARY:
This is a Python class implementation of an Automatic Speech Recognition (ASR) system using the Kaldi framework. It appears to be part of a larger library or application for speech recognition and post-processing.

The class has several methods:

1. `__init__`: Initializes the ASR system with various parameters, such as the model configuration, input streams, and output types.
2. `run_transcription`: Performs ASR transcription on a given audio input using a pre-trained model or a custom model.
3. `postprocess`: Post-processes the ASR output by decoding the transcriptions, handling timestamps and language information, and reformatting the output.

The class seems to be designed for flexibility and customization, with options to use various models, decoding algorithms, and output formats.

Some notable features of this implementation include:

* Integration with the Kaldi framework for speech recognition
* Support for multiple ASR models, including pre-trained and custom models
* Flexible output formats, including text-based transcripts and timestamped data
* Customization options for decoding algorithms and language handling
* Error detection and handling mechanisms for post-processing errors


key: chroma:document
string_value:
CODE:

def _find_timestamp_sequence(sequences, tokenizer, feature_extractor, max_source_positions):
    """
    Computes the final sequences by merging the end of the nth sequence with the beginning of the n+1th sequence. Since
    `WhisperForConditionalGeneration` produces the timestamps pairwise, we filter the consecutive timestamps and only
    iterate over them. We keep track of the `time` which indicates the actual starting time of the chunk that is
    processed. We need to make sure to offset the timestamps tokens by the `time` in order for the tokenizer to
    properly compute the final `offset`.
    """
    # index of the first timestamp token
    timestamp_begin = tokenizer.convert_tokens_to_ids("<|notimestamps|>") + 1
    items = []
    # approximation of the token to time ratio : ~0.2seconds
    time_precision = feature_extractor.chunk_length / max_source_positions
    time = 0
    for seq_idx, item in enumerate(sequences):
        sequence, stride = item
        if isinstance(sequence, list):
            sequence = np.array(sequence)
        chunk_len, stride_left, stride_right = stride
        sequence = sequence.squeeze(0)
        # get rid of the `forced_decoder_idx` that are use to parametrize the generation
        begin_idx = np.where(sequence == timestamp_begin)[0][0] if timestamp_begin in sequence else 0
        sequence = sequence[begin_idx:]

        timestamp_tokens = sequence >= timestamp_begin
        if seq_idx != 0 and sum(timestamp_tokens) > 0:
            consecutive = np.where(timestamp_tokens[:-1] & timestamp_tokens[1:])[0] + 1
            last_timestamp = np.where(timestamp_tokens)[0][-1]
            consecutive = np.append(consecutive, last_timestamp) if last_timestamp not in consecutive else consecutive
            time -= stride_left + stride_right
            offset = int((time / feature_extractor.sampling_rate) / time_precision)
            overlap_time = int((stride_left / feature_extractor.sampling_rate) / time_precision)
            # relevant timestamps are in the overlapping part
            relevant_timestamp = np.where(sequence[consecutive] >= timestamp_begin + overlap_time)[0]
            if relevant_timestamp.shape[0] > 0:
                relevant_timestamp = (
                    consecutive[relevant_timestamp[0] - 1] if relevant_timestamp[0] > 0 else consecutive[0]
                )
                # if a big stride is used, we need to check some of the previous items for the best overlap
                best_match = 0
                sliced_sequence = []
                for idx, previous_sequence in enumerate(reversed(items)):
                    previous_tokens = previous_sequence[1:-1]
                    if previous_sequence[0] < (timestamp_begin + offset - overlap_time) and idx != 0:
                        break  # the previous sequence is too far in the past
                    if len(previous_tokens) > 0:
                        # find the longest common sequence between the overlapping parts
                        index_left, index_right, match_length = _fast_find_longest_common_sequence(
                            sequence[1:relevant_timestamp], previous_tokens
                        )
                        # don't do anything if only 1 token was matched
                        if match_length > 1 and match_length > best_match:
                            best_match = match_length
                            best_idx = idx
                            end_of_curr_sequence_idx = (
                                np.where(sequence[index_left + 1 :] >= timestamp_begin)[0][0] + 1
                            )
                            end_of_curr_sequence_idx = end_of_curr_sequence_idx + 1 + index_left
                            # if all the tokens are matched, suffix
                            if index_left == 0 and match_length == len(previous_tokens):
                                sliced_sequence = np.insert(
                                    sequence[index_left + 1 : end_of_curr_sequence_idx], 0, previous_sequence[0]
                                )
                                sliced_sequence[-1] = previous_sequence[-1]
                            # if part of the previous sequence is not taken
                            elif index_left >= 0:
                                sliced_sequence = sequence[index_left + 1 : end_of_curr_sequence_idx]
                                # let's insert the missing part of the previous sequence
                                previous_slice = (
                                    previous_sequence[: index_right + 1] if index_right > 0 else [previous_sequence[0]]
                                )
                                sliced_sequence = np.insert(sliced_sequence, 0, previous_slice)
                                sliced_sequence[-1] += offset

                if len(sliced_sequence) > 0:
                    items[len(items) - best_idx - 1] = sliced_sequence
                    items = items[: len(items) - best_idx]
                    sequence = sequence[end_of_curr_sequence_idx:]

        # sequence might have changed
        timestamp_tokens = sequence >= timestamp_begin
        consecutive = np.where(timestamp_tokens[:-1] & timestamp_tokens[1:])[0] + 1
        if sum(timestamp_tokens) > 0:
            last_timestamp = np.where(timestamp_tokens)[0][-1]
            consecutive = (
                np.append(consecutive, last_timestamp + 1) if last_timestamp not in consecutive else consecutive
            )

        if len(consecutive) > 0:
            last_slice = 0
            for current_slice in consecutive:
                actual_offset = items[-1][-1] if seq_idx != 0 or last_slice != 0 else sequence[0]
                sliced_tokens = sequence[last_slice:current_slice]
                duration = sliced_tokens[-1] - sliced_tokens[0]
                sliced_tokens[0] = actual_offset
                sliced_tokens[-1] = actual_offset + duration
                items.append(sliced_tokens)
                last_slice = current_slice

        time += chunk_len
    result = []
    for i in range(len(items)):
        result += items[i].tolist()


SUMMARY:
**Function Summary: `_find_timestamp_sequence`**

**Main Purpose:** Merges timestamp sequences by finding overlapping parts between consecutive timestamps.

**Primary Functionality:**

1. Computes final sequences by merging end of nth sequence with beginning of n+1th sequence, offsetting tokens by time.
2. Filters out non-consecutive timestamps and finds overlapping parts.
3. Finds longest common sequences between overlapping parts using `_fast_find_longest_common_sequence`.
4. Inserts matching token to overlap and adjusts token offsets accordingly.

**Key Design Choices:**

* Uses `concurrent` feature extraction for timestamp sequence merging.
* Inverts `timestamp_begin` index value for correct lookup.
* Utilizes `time_precision` approximation when finding overlapping parts.

**Dependences/Requirements:**

* `tokenizer`
* `feature_extractor` class with:
	+ `sampling_rate`
	+ `chunk_length`
* `_fast_find_longest_common_sequence` function (not provided in the code snippet).

**Data Structures and Variables:**

* `sequences`: input list of timestamp sequences.
* `items`: output list of merged timestamp sequences.
* `timestamp_begin`: index value for converting timestamps to IDs.
* `time`: current time offset.

Note that this summary captures the main purpose, key design choices, dependencies, and data structures used in the function, while omitting specific implementation details that may not add value to understanding the overall functionality.


key: chroma:document
string_value:
CODE:

def get_supported_tasks() -> List[str]:
    """
    Returns a list of supported task strings.
    """


SUMMARY:
**Function Summary**

*   `get_supported_tasks()` Function
    *   Purpose: Retrieves and returns a list of supported tasks as task strings from the `pipeline_registry` using the `get_supported_tasks()` method on it.
*   Implementation Details:
    *   The function utilizes the `PIPELINE_REGISTRY` class to retrieve a list of supported tasks by calling its `get_supported_tasks()` method.
        +   Interface: `get_supported_tasks() -> List[str]`
        +   Method: `get_supported_tasks()`
*   Dependencies:
    *   **External Dependency:** None
    *   **Class/Function Reference:** `PIPELINE_REGISTRY` (a class that provides registry functionality)

**Summary in RAG Format:** 

R|---`get_supported_tasks`: Retrieves and returns a list of supported tasks as task strings from the `pipeline_registry`.|
A|  Implementation Details: Uses `pipeline_REGISTRY.get_supported_tasks()` to retrieve supported tasks.|
I|  Dependencies: `PIPELINE Registry` class |
D|  Signature: `get_supported_tasks() -> List[str]`


key: chroma:document
string_value:
CODE:

def get_task(model: str, token: Optional[str] = None, **deprecated_kwargs) -> str:
    use_auth_token = deprecated_kwargs.pop("use_auth_token", None)
    if use_auth_token is not None:
        warnings.warn(
            "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
            FutureWarning,
        )
        if token is not None:
            raise ValueError("`token` and `use_auth_token` are both specified. Please set only the argument `token`.")
        token = use_auth_token

    if is_offline_mode():
        raise RuntimeError("You cannot infer task automatically within `pipeline` when using offline mode")
    try:
        info = model_info(model, token=token)
    except Exception as e:
        raise RuntimeError(f"Instantiating a pipeline without a task set raised an error: {e}")
    if not info.pipeline_tag:
        raise RuntimeError(
            f"The model {model} does not seem to have a correct `pipeline_tag` set to infer the task automatically"
        )
    if getattr(info, "library_name", "transformers") not in {"transformers", "timm"}:
        raise RuntimeError(f"This model is meant to be used with {info.library_name} not with transformers")
    task = info.pipeline_tag


SUMMARY:
**Concise Summary:**

The `get_task` function retrieves and returns the pipeline tag for a given model. It:

1. Validates input parameters, specifically handling deprecated arguments and token usage.
2. Checks if the model is running in offline mode and raises an error if so.
3. Tries to instantiate a pipeline without manually setting the task, and catches any exceptions.
4. Verifies that the library name matches "transformers" or "timm", raising an error otherwise.
5. Returns the pipeline tag, which is assumed to be set in the model's configuration.

**Key Implementation Details:**

* Handles deprecated `use_auth_token` argument and warns about potential removal.
* Checks for offline mode and raises error if applicable.
* Verifies library name consistency before proceeding.

**Notable Dependencies/Requirements:**

* `token`: an optional string parameter used as authentication token.
* `deprecated_kwargs`: a dictionary containing deprecated arguments (e.g., `use_auth_token`).
* `model_info`, `is_offline_mode`, and `warnings` modules are not explicitly imported, but implied for their usage.


key: chroma:document
string_value:
CODE:

def check_task(task: str) -> Tuple[str, Dict, Any]:
    """
    Checks an incoming task string, to validate it's correct and return the default Pipeline and Model classes, and
    default models if they exist.

    Args:
        task (`str`):
            The task defining which pipeline will be returned. Currently accepted tasks are:

            - `"audio-classification"`
            - `"automatic-speech-recognition"`
            - `"conversational"`
            - `"depth-estimation"`
            - `"document-question-answering"`
            - `"feature-extraction"`
            - `"fill-mask"`
            - `"image-classification"`
            - `"image-feature-extraction"`
            - `"image-segmentation"`
            - `"image-to-text"`
            - `"image-to-image"`
            - `"object-detection"`
            - `"question-answering"`
            - `"summarization"`
            - `"table-question-answering"`
            - `"text2text-generation"`
            - `"text-classification"` (alias `"sentiment-analysis"` available)
            - `"text-generation"`
            - `"text-to-audio"` (alias `"text-to-speech"` available)
            - `"token-classification"` (alias `"ner"` available)
            - `"translation"`
            - `"translation_xx_to_yy"`
            - `"video-classification"`
            - `"visual-question-answering"` (alias `"vqa"` available)
            - `"zero-shot-classification"`
            - `"zero-shot-image-classification"`
            - `"zero-shot-object-detection"`

    Returns:
        (normalized_task: `str`, task_defaults: `dict`, task_options: (`tuple`, None)) The normalized task name
        (removed alias and options). The actual dictionary required to initialize the pipeline and some extra task
        options for parametrized tasks like "translation_XX_to_YY"


    """


SUMMARY:
**Function Summary: `check_task`**

**Main Purpose:** Validates an incoming task string against predefined tasks and returns the default Pipeline and Model classes along with extra options for parametrized tasks.

**Key Implementation Details:**

* The function uses a predefined `PIPELINE_REGISTRY` object to perform validation.
* It checks if the input `task` is one of the accepted tasks and normalizes it by removing aliases.
* Returns a tuple containing:
	+ `normalized_task`: the normalized task name
	+ `task_defaults`: a dictionary required to initialize the pipeline
	+ `task_options`: an optional string with extra options for parametrized tasks

**Notable Dependencies/Requirements:**

* `PIPELINE_REGISTRY` class, not defined in this snippet
* `Pipeline`, `Model`, and other related classes
* External dependencies not listed, assuming they are included in the `PIPELINE_REGISTRY` or elsewhere.


key: chroma:document
string_value:
CODE:

def clean_custom_task(task_info):
    import transformers

    if "impl" not in task_info:
        raise RuntimeError("This model introduces a custom pipeline without specifying its implementation.")
    pt_class_names = task_info.get("pt", ())
    if isinstance(pt_class_names, str):
        pt_class_names = [pt_class_names]
    task_info["pt"] = tuple(getattr(transformers, c) for c in pt_class_names)
    tf_class_names = task_info.get("tf", ())
    if isinstance(tf_class_names, str):
        tf_class_names = [tf_class_names]
    task_info["tf"] = tuple(getattr(transformers, c) for c in tf_class_names)


SUMMARY:
**Function Summary:**

`clean_custom_task` function cleans and normalizes custom task information for use with the Hugging Face Transformers library.

**Main Purpose and Functionality:**

Checks if a custom pipeline implementation is specified and updates the `task_info` dictionary to include necessary transformers classes.

**Key Implementation Details:**

* Handles cases where pipeline names are passed as strings
* Uses `getattr(transformers, c)` to dynamically import transformer classes
* Returns updated `task_info` with loaded transformer classes

**Notable Dependencies or Requirements:**

* Hugging Face Transformers library (`transformers`)
* Python runtime environment


key: chroma:document
string_value:
CODE:

def pipeline(
    task: str = None,
    model: Optional[Union[str, "PreTrainedModel", "TFPreTrainedModel"]] = None,
    config: Optional[Union[str, PretrainedConfig]] = None,
    tokenizer: Optional[Union[str, PreTrainedTokenizer, "PreTrainedTokenizerFast"]] = None,
    feature_extractor: Optional[Union[str, PreTrainedFeatureExtractor]] = None,
    image_processor: Optional[Union[str, BaseImageProcessor]] = None,
    processor: Optional[Union[str, ProcessorMixin]] = None,
    framework: Optional[str] = None,
    revision: Optional[str] = None,
    use_fast: bool = True,
    token: Optional[Union[str, bool]] = None,
    device: Optional[Union[int, str, "torch.device"]] = None,
    device_map=None,
    torch_dtype=None,
    trust_remote_code: Optional[bool] = None,
    model_kwargs: Dict[str, Any] = None,
    pipeline_class: Optional[Any] = None,
    **kwargs,
) -> Pipeline:
    """
    Utility factory method to build a [`Pipeline`].

    A pipeline consists of:

        - One or more components for pre-processing model inputs, such as a [tokenizer](tokenizer),
        [image_processor](image_processor), [feature_extractor](feature_extractor), or [processor](processors).
        - A [model](model) that generates predictions from the inputs.
        - Optional post-processing steps to refine the model's output, which can also be handled by processors.

    <Tip>
    While there are such optional arguments as `tokenizer`, `feature_extractor`, `image_processor`, and `processor`,
    they shouldn't be specified all at once. If these components are not provided, `pipeline` will try to load
    required ones automatically. In case you want to provide these components explicitly, please refer to a
    specific pipeline in order to get more details regarding what components are required.
    </Tip>

    Args:
        task (`str`):
            The task defining which pipeline will be returned. Currently accepted tasks are:

            - `"audio-classification"`: will return a [`AudioClassificationPipeline`].
            - `"automatic-speech-recognition"`: will return a [`AutomaticSpeechRecognitionPipeline`].
            - `"depth-estimation"`: will return a [`DepthEstimationPipeline`].
            - `"document-question-answering"`: will return a [`DocumentQuestionAnsweringPipeline`].
            - `"feature-extraction"`: will return a [`FeatureExtractionPipeline`].
            - `"fill-mask"`: will return a [`FillMaskPipeline`]:.
            - `"image-classification"`: will return a [`ImageClassificationPipeline`].
            - `"image-feature-extraction"`: will return an [`ImageFeatureExtractionPipeline`].
            - `"image-segmentation"`: will return a [`ImageSegmentationPipeline`].
            - `"image-text-to-text"`: will return a [`ImageTextToTextPipeline`].
            - `"image-to-image"`: will return a [`ImageToImagePipeline`].
            - `"image-to-text"`: will return a [`ImageToTextPipeline`].
            - `"mask-generation"`: will return a [`MaskGenerationPipeline`].
            - `"object-detection"`: will return a [`ObjectDetectionPipeline`].
            - `"question-answering"`: will return a [`QuestionAnsweringPipeline`].
            - `"summarization"`: will return a [`SummarizationPipeline`].
            - `"table-question-answering"`: will return a [`TableQuestionAnsweringPipeline`].
            - `"text2text-generation"`: will return a [`Text2TextGenerationPipeline`].
            - `"text-classification"` (alias `"sentiment-analysis"` available): will return a
              [`TextClassificationPipeline`].
            - `"text-generation"`: will return a [`TextGenerationPipeline`]:.
            - `"text-to-audio"` (alias `"text-to-speech"` available): will return a [`TextToAudioPipeline`]:.
            - `"token-classification"` (alias `"ner"` available): will return a [`TokenClassificationPipeline`].
            - `"translation"`: will return a [`TranslationPipeline`].
            - `"translation_xx_to_yy"`: will return a [`TranslationPipeline`].
            - `"video-classification"`: will return a [`VideoClassificationPipeline`].
            - `"visual-question-answering"`: will return a [`VisualQuestionAnsweringPipeline`].
            - `"zero-shot-classification"`: will return a [`ZeroShotClassificationPipeline`].
            - `"zero-shot-image-classification"`: will return a [`ZeroShotImageClassificationPipeline`].
            - `"zero-shot-audio-classification"`: will return a [`ZeroShotAudioClassificationPipeline`].
            - `"zero-shot-object-detection"`: will return a [`ZeroShotObjectDetectionPipeline`].

        model (`str` or [`PreTrainedModel`] or [`TFPreTrainedModel`], *optional*):
            The model that will be used by the pipeline to make predictions. This can be a model identifier or an
            actual instance of a pretrained model inheriting from [`PreTrainedModel`] (for PyTorch) or
            [`TFPreTrainedModel`] (for TensorFlow).

            If not provided, the default for the `task` will be loaded.
        config (`str` or [`PretrainedConfig`], *optional*):
            The configuration that will be used by the pipeline to instantiate the model. This can be a model
            identifier or an actual pretrained model configuration inheriting from [`PretrainedConfig`].

            If not provided, the default configuration file for the requested model will be used. That means that if
            `model` is given, its default configuration will be used. However, if `model` is not supplied, this
            `task`'s default model's config is used instead.
        tokenizer (`str` or [`PreTrainedTokenizer`], *optional*):
            The tokenizer that will be used by the pipeline to encode data for the model. This can be a model
            identifier or an actual pretrained tokenizer inheriting from [`PreTrainedTokenizer`].

            If not provided, the default tokenizer for the given `model` will be loaded (if it is a string). If `model`
            is not specified or not a string, then the default tokenizer for `config` is loaded (if it is a string).
            However, if `config` is also not given or not a string, then the default tokenizer for the given `task`
            will be loaded.
        feature_extractor (`str` or [`PreTrainedFeatureExtractor`], *optional*):
            The feature extractor that will be used by the pipeline to encode data for the model. This can be a model
            identifier or an actual pretrained feature extractor inheriting from [`PreTrainedFeatureExtractor`].

            Feature extractors are used for non-NLP models, such as Speech or Vision models as well as multi-modal
            models. Multi-modal models will also require a tokenizer to be passed.

            If not provided, the default feature extractor for the given `model` will be loaded (if it is a string). If
            `model` is not specified or not a string, then the default feature extractor for `config` is loaded (if it
            is a string). However, if `config` is also not given or not a string, then the default feature extractor
            for the given `task` will be loaded.
        image_processor (`str` or [`BaseImageProcessor`], *optional*):
            The image processor that will be used by the pipeline to preprocess images for the model. This can be a
            model identifier or an actual image processor inheriting from [`BaseImageProcessor`].

            Image processors are used for Vision models and multi-modal models that require image inputs. Multi-modal
            models will also require a tokenizer to be passed.

            If not provided, the default image processor for the given `model` will be loaded (if it is a string). If
            `model` is not specified or not a string, then the default image processor for `config` is loaded (if it is
            a string).
        processor (`str` or [`ProcessorMixin`], *optional*):
            The processor that will be used by the pipeline to preprocess data for the model. This can be a model
            identifier or an actual processor inheriting from [`ProcessorMixin`].

            Processors are used for multi-modal models that require multi-modal inputs, for example, a model that
            requires both text and image inputs.

            If not provided, the default processor for the given `model` will be loaded (if it is a string). If `model`
            is not specified or not a string, then the default processor for `config` is loaded (if it is a string).
        framework (`str`, *optional*):
            The framework to use, either `"pt"` for PyTorch or `"tf"` for TensorFlow. The specified framework must be
            installed.

            If no framework is specified, will default to the one currently installed. If no framework is specified and
            both frameworks are installed, will default to the framework of the `model`, or to PyTorch if no model is
            provided.
        revision (`str`, *optional*, defaults to `"main"`):
            When passing a task name or a string model identifier: The specific model version to use. It can be a
            branch name, a tag name, or a commit id, since we use a git-based system for storing models and other
            artifacts on huggingface.co, so `revision` can be any identifier allowed by git.
        use_fast (`bool`, *optional*, defaults to `True`):
            Whether or not to use a Fast tokenizer if possible (a [`PreTrainedTokenizerFast`]).
        use_auth_token (`str` or *bool*, *optional*):
            The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
            when running `huggingface-cli login` (stored in `~/.huggingface`).
        device (`int` or `str` or `torch.device`):
            Defines the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank like `1`) on which this
            pipeline will be allocated.
        device_map (`str` or `Dict[str, Union[int, str, torch.device]`, *optional*):
            Sent directly as `model_kwargs` (just a simpler shortcut). When `accelerate` library is present, set
            `device_map="auto"` to compute the most optimized `device_map` automatically (see
            [here](https://huggingface.co/docs/accelerate/main/en/package_reference/big_modeling#accelerate.cpu_offload)
            for more information).

            <Tip warning={true}>

            Do not use `device_map` AND `device` at the same time as they will conflict

            </Tip>

        torch_dtype (`str` or `torch.dtype`, *optional*):
            Sent directly as `model_kwargs` (just a simpler shortcut) to use the available precision for this model
            (`torch.float16`, `torch.bfloat16`, ... or `"auto"`).
        trust_remote_code (`bool`, *optional*, defaults to `False`):
            Whether or not to allow for custom code defined on the Hub in their own modeling, configuration,
            tokenization or even pipeline files. This option should only be set to `True` for repositories you trust
            and in which you have read the code, as it will execute code present on the Hub on your local machine.
        model_kwargs (`Dict[str, Any]`, *optional*):
            Additional dictionary of keyword arguments passed along to the model's `from_pretrained(...,
            **model_kwargs)` function.
        kwargs (`Dict[str, Any]`, *optional*):
            Additional keyword arguments passed along to the specific pipeline init (see the documentation for the
            corresponding pipeline class for possible values).

    Returns:
        [`Pipeline`]: A suitable pipeline for the task.

    Examples:

    ```python
    >>> from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer

    >>> # Sentiment analysis pipeline
    >>> analyzer = pipeline("sentiment-analysis")

    >>> # Question answering pipeline, specifying the checkpoint identifier
    >>> oracle = pipeline(
    ...     "question-answering", model="distilbert/distilbert-base-cased-distilled-squad", tokenizer="google-bert/bert-base-cased"
    ... )

    >>> # Named entity recognition pipeline, passing in a specific model and tokenizer
    >>> model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
    >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
    >>> recognizer = pipeline("ner", model=model, tokenizer=tokenizer)
    ```"""
    if model_kwargs is None:
        model_kwargs = {}
    # Make sure we only pass use_auth_token once as a kwarg (it used to be possible to pass it in model_kwargs,
    # this is to keep BC).
    use_auth_token = model_kwargs.pop("use_auth_token", None)
    if use_auth_token is not None:
        warnings.warn(
            "The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.",
            FutureWarning,
        )
        if token is not None:
            raise ValueError("`token` and `use_auth_token` are both specified. Please set only the argument `token`.")
        token = use_auth_token

    code_revision = kwargs.pop("code_revision", None)
    commit_hash = kwargs.pop("_commit_hash", None)

    hub_kwargs = {
        "revision": revision,
        "token": token,
        "trust_remote_code": trust_remote_code,
        "_commit_hash": commit_hash,
    }

    if task is None and model is None:
        raise RuntimeError(
            "Impossible to instantiate a pipeline without either a task or a model "
            "being specified. "
            "Please provide a task class or a model"
        )

    if model is None and tokenizer is not None:
        raise RuntimeError(
            "Impossible to instantiate a pipeline with tokenizer specified but not the model as the provided tokenizer"
            " may not be compatible with the default model. Please provide a PreTrainedModel class or a"
            " path/identifier to a pretrained model when providing tokenizer."
        )
    if model is None and feature_extractor is not None:
        raise RuntimeError(
            "Impossible to instantiate a pipeline with feature_extractor specified but not the model as the provided"
            " feature_extractor may not be compatible with the default model. Please provide a PreTrainedModel class"
            " or a path/identifier to a pretrained model when providing feature_extractor."
        )
    if isinstance(model, Path):
        model = str(model)

    if commit_hash is None:
        pretrained_model_name_or_path = None
        if isinstance(config, str):
            pretrained_model_name_or_path = config
        elif config is None and isinstance(model, str):
            pretrained_model_name_or_path = model

        if not isinstance(config, PretrainedConfig) and pretrained_model_name_or_path is not None:
            # We make a call to the config file first (which may be absent) to get the commit hash as soon as possible
            resolved_config_file = cached_file(
                pretrained_model_name_or_path,
                CONFIG_NAME,
                _raise_exceptions_for_gated_repo=False,
                _raise_exceptions_for_missing_entries=False,
                _raise_exceptions_for_connection_errors=False,
                cache_dir=model_kwargs.get("cache_dir"),
                **hub_kwargs,
            )
            hub_kwargs["_commit_hash"] = extract_commit_hash(resolved_config_file, commit_hash)
        else:
            hub_kwargs["_commit_hash"] = getattr(config, "_commit_hash", None)

    # Config is the primordial information item.
    # Instantiate config if needed
    if isinstance(config, str):
        config = AutoConfig.from_pretrained(
            config, _from_pipeline=task, code_revision=code_revision, **hub_kwargs, **model_kwargs
        )
        hub_kwargs["_commit_hash"] = config._commit_hash
    elif config is None and isinstance(model, str):
        # Check for an adapter file in the model path if PEFT is available
        if is_peft_available():
            # `find_adapter_config_file` doesn't accept `trust_remote_code`
            _hub_kwargs = {k: v for k, v in hub_kwargs.items() if k != "trust_remote_code"}
            maybe_adapter_path = find_adapter_config_file(
                model,
                token=hub_kwargs["token"],
                revision=hub_kwargs["revision"],
                _commit_hash=hub_kwargs["_commit_hash"],
            )

            if maybe_adapter_path is not None:
                with open(maybe_adapter_path, "r", encoding="utf-8") as f:
                    adapter_config = json.load(f)
                    model = adapter_config["base_model_name_or_path"]

        config = AutoConfig.from_pretrained(
            model, _from_pipeline=task, code_revision=code_revision, **hub_kwargs, **model_kwargs
        )
        hub_kwargs["_commit_hash"] = config._commit_hash

    custom_tasks = {}
    if config is not None and len(getattr(config, "custom_pipelines", {})) > 0:
        custom_tasks = config.custom_pipelines
        if task is None and trust_remote_code is not False:
            if len(custom_tasks) == 1:
                task = list(custom_tasks.keys())[0]
            else:
                raise RuntimeError(
                    "We can't infer the task automatically for this model as there are multiple tasks available. Pick "
                    f"one in {', '.join(custom_tasks.keys())}"
                )

    if task is None and model is not None:
        if not isinstance(model, str):
            raise RuntimeError(
                "Inferring the task automatically requires to check the hub with a model_id defined as a `str`. "
                f"{model} is not a valid model_id."
            )
        task = get_task(model, token)

    # Retrieve the task
    if task in custom_tasks:
        normalized_task = task
        targeted_task, task_options = clean_custom_task(custom_tasks[task])
        if pipeline_class is None:
            if not trust_remote_code:
                raise ValueError(
                    "Loading this pipeline requires you to execute the code in the pipeline file in that"
                    " repo on your local machine. Make sure you have read the code there to avoid malicious use, then"
                    " set the option `trust_remote_code=True` to remove this error."
                )
            class_ref = targeted_task["impl"]
            pipeline_class = get_class_from_dynamic_module(
                class_ref,
                model,
                code_revision=code_revision,
                **hub_kwargs,
            )
    else:
        normalized_task, targeted_task, task_options = check_task(task)
        if pipeline_class is None:
            pipeline_class = targeted_task["impl"]

    # Use default model/config/tokenizer for the task if no model is provided
    if model is None:
        # At that point framework might still be undetermined
        model, default_revision = get_default_model_and_revision(targeted_task, framework, task_options)
        revision = revision if revision is not None else default_revision
        logger.warning(
            f"No model was supplied, defaulted to {model} and revision"
            f" {revision} ({HUGGINGFACE_CO_RESOLVE_ENDPOINT}/{model}).\n"
            "Using a pipeline without specifying a model name and revision in production is not recommended."
        )
        hub_kwargs["revision"] = revision
        if config is None and isinstance(model, str):
            config = AutoConfig.from_pretrained(model, _from_pipeline=task, **hub_kwargs, **model_kwargs)
            hub_kwargs["_commit_hash"] = config._commit_hash

    if device_map is not None:
        if "device_map" in model_kwargs:
            raise ValueError(
                'You cannot use both `pipeline(... device_map=..., model_kwargs={"device_map":...})` as those'
                " arguments might conflict, use only one.)"
            )
        if device is not None:
            logger.warning(
                "Both `device` and `device_map` are specified. `device` will override `device_map`. You"
                " will most likely encounter unexpected behavior. Please remove `device` and keep `device_map`."
            )
        model_kwargs["device_map"] = device_map
    if torch_dtype is not None:
        if "torch_dtype" in model_kwargs:
            raise ValueError(
                'You cannot use both `pipeline(... torch_dtype=..., model_kwargs={"torch_dtype":...})` as those'
                " arguments might conflict, use only one.)"
            )
        if isinstance(torch_dtype, str) and hasattr(torch, torch_dtype):
            torch_dtype = getattr(torch, torch_dtype)
        model_kwargs["torch_dtype"] = torch_dtype

    model_name = model if isinstance(model, str) else None

    # Load the correct model if possible
    # Infer the framework from the model if not already defined
    if isinstance(model, str) or framework is None:
        model_classes = {"tf": targeted_task["tf"], "pt": targeted_task["pt"]}
        framework, model = infer_framework_load_model(
            model,
            model_classes=model_classes,
            config=config,
            framework=framework,
            task=task,
            **hub_kwargs,
            **model_kwargs,
        )

    model_config = model.config
    hub_kwargs["_commit_hash"] = model.config._commit_hash

    load_tokenizer = type(model_config) in TOKENIZER_MAPPING or model_config.tokenizer_class is not None
    load_feature_extractor = type(model_config) in FEATURE_EXTRACTOR_MAPPING or feature_extractor is not None
    load_image_processor = type(model_config) in IMAGE_PROCESSOR_MAPPING or image_processor is not None
    load_processor = type(model_config) in PROCESSOR_MAPPING or processor is not None

    # Check that pipeline class required loading
    load_tokenizer = load_tokenizer and pipeline_class._load_tokenizer
    load_feature_extractor = load_feature_extractor and pipeline_class._load_feature_extractor
    load_image_processor = load_image_processor and pipeline_class._load_image_processor
    load_processor = load_processor and pipeline_class._load_processor

    # If `model` (instance of `PretrainedModel` instead of `str`) is passed (and/or same for config), while
    # `image_processor` or `feature_extractor` is `None`, the loading will fail. This happens particularly for some
    # vision tasks when calling `pipeline()` with `model` and only one of the `image_processor` and `feature_extractor`.
    # TODO: we need to make `NO_IMAGE_PROCESSOR_TASKS` and `NO_FEATURE_EXTRACTOR_TASKS` more robust to avoid such issue.
    # This block is only temporarily to make CI green.
    if load_image_processor and load_feature_extractor:
        load_feature_extractor = False

    if (
        tokenizer is None
        and not load_tokenizer
        and normalized_task not in NO_TOKENIZER_TASKS
        # Using class name to avoid importing the real class.
        and (
            model_config.__class__.__name__ in MULTI_MODEL_AUDIO_CONFIGS
            or model_config.__class__.__name__ in MULTI_MODEL_VISION_CONFIGS
        )
    ):
        # This is a special category of models, that are fusions of multiple models
        # so the model_config might not define a tokenizer, but it seems to be
        # necessary for the task, so we're force-trying to load it.
        load_tokenizer = True
    if (
        image_processor is None
        and not load_image_processor
        and normalized_task not in NO_IMAGE_PROCESSOR_TASKS
        # Using class name to avoid importing the real class.
        and model_config.__class__.__name__ in MULTI_MODEL_VISION_CONFIGS
    ):
        # This is a special category of models, that are fusions of multiple models
        # so the model_config might not define a tokenizer, but it seems to be
        # necessary for the task, so we're force-trying to load it.
        load_image_processor = True
    if (
        feature_extractor is None
        and not load_feature_extractor
        and normalized_task not in NO_FEATURE_EXTRACTOR_TASKS
        # Using class name to avoid importing the real class.
        and model_config.__class__.__name__ in MULTI_MODEL_AUDIO_CONFIGS
    ):
        # This is a special category of models, that are fusions of multiple models
        # so the model_config might not define a tokenizer, but it seems to be
        # necessary for the task, so we're force-trying to load it.
        load_feature_extractor = True

    if task in NO_TOKENIZER_TASKS:
        # These will never require a tokenizer.
        # the model on the other hand might have a tokenizer, but
        # the files could be missing from the hub, instead of failing
        # on such repos, we just force to not load it.
        load_tokenizer = False

    if task in NO_FEATURE_EXTRACTOR_TASKS:
        load_feature_extractor = False
    if task in NO_IMAGE_PROCESSOR_TASKS:
        load_image_processor = False

    if load_tokenizer:
        # Try to infer tokenizer from model or config name (if provided as str)
        if tokenizer is None:
            if isinstance(model_name, str):
                tokenizer = model_name
            elif isinstance(config, str):
                tokenizer = config
            else:
                # Impossible to guess what is the right tokenizer here
                raise Exception(
                    "Impossible to guess which tokenizer to use. "
                    "Please provide a PreTrainedTokenizer class or a path/identifier to a pretrained tokenizer."
                )

        # Instantiate tokenizer if needed
        if isinstance(tokenizer, (str, tuple)):
            if isinstance(tokenizer, tuple):
                # For tuple we have (tokenizer name, {kwargs})
                use_fast = tokenizer[1].pop("use_fast", use_fast)
                tokenizer_identifier = tokenizer[0]
                tokenizer_kwargs = tokenizer[1]
            else:
                tokenizer_identifier = tokenizer
                tokenizer_kwargs = model_kwargs.copy()
                tokenizer_kwargs.pop("torch_dtype", None)

            tokenizer = AutoTokenizer.from_pretrained(
                tokenizer_identifier, use_fast=use_fast, _from_pipeline=task, **hub_kwargs, **tokenizer_kwargs
            )

    if load_image_processor:
        # Try to infer image processor from model or config name (if provided as str)
        if image_processor is None:
            if isinstance(model_name, str):
                image_processor = model_name
            elif isinstance(config, str):
                image_processor = config
            # Backward compatibility, as `feature_extractor` used to be the name
            # for `ImageProcessor`.
            elif feature_extractor is not None and isinstance(feature_extractor, BaseImageProcessor):
                image_processor = feature_extractor
            else:
                # Impossible to guess what is the right image_processor here
                raise Exception(
                    "Impossible to guess which image processor to use. "
                    "Please provide a PreTrainedImageProcessor class or a path/identifier "
                    "to a pretrained image processor."
                )

        # Instantiate image_processor if needed
        if isinstance(image_processor, (str, tuple)):
            image_processor = AutoImageProcessor.from_pretrained(
                image_processor, _from_pipeline=task, **hub_kwargs, **model_kwargs
            )

    if load_feature_extractor:
        # Try to infer feature extractor from model or config name (if provided as str)
        if feature_extractor is None:
            if isinstance(model_name, str):
                feature_extractor = model_name
            elif isinstance(config, str):
                feature_extractor = config
            else:
                # Impossible to guess what is the right feature_extractor here
                raise Exception(
                    "Impossible to guess which feature extractor to use. "
                    "Please provide a PreTrainedFeatureExtractor class or a path/identifier "
                    "to a pretrained feature extractor."
                )

        # Instantiate feature_extractor if needed
        if isinstance(feature_extractor, (str, tuple)):
            feature_extractor = AutoFeatureExtractor.from_pretrained(
                feature_extractor, _from_pipeline=task, **hub_kwargs, **model_kwargs
            )

            if (
                feature_extractor._processor_class
                and feature_extractor._processor_class.endswith("WithLM")
                and isinstance(model_name, str)
            ):
                try:
                    import kenlm  # to trigger `ImportError` if not installed
                    from pyctcdecode import BeamSearchDecoderCTC

                    if os.path.isdir(model_name) or os.path.isfile(model_name):
                        decoder = BeamSearchDecoderCTC.load_from_dir(model_name)
                    else:
                        language_model_glob = os.path.join(
                            BeamSearchDecoderCTC._LANGUAGE_MODEL_SERIALIZED_DIRECTORY, "*"
                        )
                        alphabet_filename = BeamSearchDecoderCTC._ALPHABET_SERIALIZED_FILENAME
                        allow_patterns = [language_model_glob, alphabet_filename]
                        decoder = BeamSearchDecoderCTC.load_from_hf_hub(model_name, allow_patterns=allow_patterns)

                    kwargs["decoder"] = decoder
                except ImportError as e:
                    logger.warning(f"Could not load the `decoder` for {model_name}. Defaulting to raw CTC. Error: {e}")
                    if not is_kenlm_available():
                        logger.warning("Try to install `kenlm`: `pip install kenlm")

                    if not is_pyctcdecode_available():
                        logger.warning("Try to install `pyctcdecode`: `pip install pyctcdecode")

    if load_processor:
        # Try to infer processor from model or config name (if provided as str)
        if processor is None:
            if isinstance(model_name, str):
                processor = model_name
            elif isinstance(config, str):
                processor = config
            else:
                # Impossible to guess what is the right processor here
                raise Exception(
                    "Impossible to guess which processor to use. "
                    "Please provide a processor instance or a path/identifier "
                    "to a processor."
                )

        # Instantiate processor if needed
        if isinstance(processor, (str, tuple)):
            processor = AutoProcessor.from_pretrained(processor, _from_pipeline=task, **hub_kwargs, **model_kwargs)
            if not isinstance(processor, ProcessorMixin):
                raise TypeError(
                    "Processor was loaded, but it is not an instance of `ProcessorMixin`. "
                    f"Got type `{type(processor)}` instead. Please check that you specified "
                    "correct pipeline task for the model and model has processor implemented and saved."
                )

    if task == "translation" and model.config.task_specific_params:
        for key in model.config.task_specific_params:
            if key.startswith("translation"):
                task = key
                warnings.warn(
                    f'"translation" task was used, instead of "translation_XX_to_YY", defaulting to "{task}"',
                    UserWarning,
                )
                break

    if tokenizer is not None:
        kwargs["tokenizer"] = tokenizer

    if feature_extractor is not None:
        kwargs["feature_extractor"] = feature_extractor

    if torch_dtype is not None:
        kwargs["torch_dtype"] = torch_dtype

    if image_processor is not None:
        kwargs["image_processor"] = image_processor

    if device is not None:
        kwargs["device"] = device

    if processor is not None:
        kwargs["processor"] = processor



SUMMARY:
Here is a concise summary of the provided code snippet:

**Overview**

This code is part of a Natural Language Processing (NLP) model loading and creation process. It defines a function `load_model` that creates an instance of the specified `pipeline_class` by parsing the task-specific configuration, loading any necessary preprocessing layers, and setting up various model parameters.

**Key Components**

1. **Preprocessing Layers**: The code loads and instantiates various preprocessing layers such as tokenizer, image processor, and feature extractor based on task specifications.
2. **Model Parameters**: It sets various model parameters including device, torch_dtype, and processor settings according to the task requirements.
3. **Return Value**: The function returns an instance of `pipeline_class` with the specified model, framework, task, and configuration.

**Notes**

* This code appears to be based on Hugging Face's Transformers library for NLP models.
* The code uses various pre-trained models and libraries such as AutoTokenizer, AutoFeatureExtractor, and BeamSearchDecoderCTC.


key: chroma:document
string_value:
CODE:

def read_video_pyav(container, indices):
    frames = []
    container.seek(0)
    start_index = indices[0]
    end_index = indices[-1]
    for i, frame in enumerate(container.decode(video=0)):
        if i > end_index:
            break
        if i >= start_index and i in indices:
            frames.append(frame)


SUMMARY:
**Function Summary:**

`read_video_pyav(container, indices)` reads video frames from PyAV container.

**Purpose:** Extract specific frames from a video stream based on provided indices and store them as RGB values.

**Key Implementation Details:**

* Uses `container.decode()` to iterate over the video frame indices.
* Filters frames within specified range (`start_index` to `end_index`) and includes only those present in `indices`.
* Converts each frame to RGB format using `to_ndarray(format="rgb24")`.

**Dependencies:**

* PyAV
* NumPy

Note: No external dependencies or interfaces are used. The summary focuses on the key functionality and algorithms used in the function.


key: chroma:document
string_value:
CODE:

def no_collate_fn(items):
    if len(items) != 1:
        raise ValueError("This collate_fn is meant to be used with batch_size=1")


SUMMARY:
**Function Summary**

`no_collate_fn`: A custom collation function that returns the only item in an iterable, or raises an error if it's not.

**Key Implementation Details**

* Raises `ValueError` if input iterable has more than one element
* Returns single-element iterable directly (i.e., No Collation)

**Notable Dependencies/Requirements**

* None referenced; uses standard Python `raise` and `return` statements


key: chroma:document
string_value:
CODE:

def _pad(items, key, padding_value, padding_side):
    batch_size = len(items)
    if isinstance(items[0][key], torch.Tensor):
        # Others include `attention_mask` etc...
        shape = items[0][key].shape
        dim = len(shape)
        if dim == 1:
            # We have a list of 1-dim torch tensors, which can be stacked without padding
            return torch.cat([item[key] for item in items], dim=0)
        if key in ["pixel_values", "image"]:
            # This is probable image so padding shouldn't be necessary
            # B, C, H, W
            return torch.cat([item[key] for item in items], dim=0)
        elif dim == 4 and key == "input_features":
            # this is probably a mel spectrogram batched
            return torch.cat([item[key] for item in items], dim=0)
        max_length = max(item[key].shape[1] for item in items)
        min_length = min(item[key].shape[1] for item in items)
        dtype = items[0][key].dtype

        if dim == 2:
            if max_length == min_length:
                # Bypass for `ImageGPT` which doesn't provide a padding value, yet
                # we can consistently pad since the size should be matching
                return torch.cat([item[key] for item in items], dim=0)
            tensor = torch.zeros((batch_size, max_length), dtype=dtype) + padding_value
        elif dim == 3:
            tensor = torch.zeros((batch_size, max_length, shape[-1]), dtype=dtype) + padding_value
        elif dim == 4:
            tensor = torch.zeros((batch_size, max_length, shape[-2], shape[-1]), dtype=dtype) + padding_value

        for i, item in enumerate(items):
            if dim == 2:
                if padding_side == "left":
                    tensor[i, -len(item[key][0]) :] = item[key][0].clone()
                else:
                    tensor[i, : len(item[key][0])] = item[key][0].clone()
            elif dim == 3:
                if padding_side == "left":
                    tensor[i, -len(item[key][0]) :, :] = item[key][0].clone()
                else:
                    tensor[i, : len(item[key][0]), :] = item[key][0].clone()
            elif dim == 4:
                if padding_side == "left":
                    tensor[i, -len(item[key][0]) :, :, :] = item[key][0].clone()
                else:
                    tensor[i, : len(item[key][0]), :, :] = item[key][0].clone()

        return tensor
    else:


SUMMARY:
**Function Summary: `_pad(items, key, padding_value, padding_side)`**

**Main Purpose and Functionality:**
This function pads or truncates the last dimension of tensors along the `key` axis to achieve a uniform length (`max_length`) in batches. It handles different tensor dimensions (1D, 2D, 3D, 4D) and applies padding according to the specified `padding_side`.

**Key Implementation Details:**

* It checks if the first item in the batch is a torch Tensor using `isinstance(items[0][key], torch.Tensor)`.
* For different tensor dimensions, it appends or prepends padding values to maintain consistency.
* It uses `torch.cat` and slicing operations (`tensor[i, ...]`) to construct the padded tensors.

**Notable Dependencies and Requirements:**

* `torch`
* `torchvision`

No external dependencies are listed as they appear to be part of the PyTorch library.


key: chroma:document
string_value:
CODE:

def pad_collate_fn(tokenizer, feature_extractor):
    # Tokenizer
    t_padding_side = None
    # Feature extractor
    f_padding_side = None
    if tokenizer is None and feature_extractor is None:
        raise ValueError("Pipeline without tokenizer or feature_extractor cannot do batching")
    if tokenizer is not None:
        if tokenizer.pad_token_id is None:
            raise ValueError(
                "Pipeline with tokenizer without pad_token cannot do batching. You can try to set it with "
                "`pipe.tokenizer.pad_token_id = model.config.eos_token_id`."
            )
        else:
            t_padding_value = tokenizer.pad_token_id
            t_padding_side = tokenizer.padding_side
    if feature_extractor is not None:
        # Feature extractor can be images, where no padding is expected
        f_padding_value = getattr(feature_extractor, "padding_value", None)
        f_padding_side = getattr(feature_extractor, "padding_side", None)

    if t_padding_side is not None and f_padding_side is not None and t_padding_side != f_padding_side:
        raise ValueError(
            f"The feature extractor, and tokenizer don't agree on padding side {t_padding_side} != {f_padding_side}"
        )
    padding_side = "right"
    if t_padding_side is not None:
        padding_side = t_padding_side
    if f_padding_side is not None:
        padding_side = f_padding_side

    def inner(items):
        keys = set(items[0].keys())
        for item in items:
            if set(item.keys()) != keys:
                raise ValueError(
                    f"The elements of the batch contain different keys. Cannot batch them ({set(item.keys())} !="
                    f" {keys})"
                )
        # input_values, input_pixels, input_ids, ...
        padded = {}
        for key in keys:
            if key in {"input_ids"}:
                # ImageGPT uses a feature extractor
                if tokenizer is None and feature_extractor is not None:
                    _padding_value = f_padding_value
                else:
                    _padding_value = t_padding_value
            elif key in {"input_values", "pixel_values", "input_features"}:
                _padding_value = f_padding_value
            elif key in {"p_mask", "special_tokens_mask"}:
                _padding_value = 1
            elif key in {"attention_mask", "token_type_ids"}:
                _padding_value = 0
            else:
                # This is likely another random key maybe even user provided
                _padding_value = 0
            padded[key] = _pad(items, key, _padding_value, padding_side)
        return padded



SUMMARY:
**Function Summary: `pad_collate_fn`**

**Purpose and Functionality:**
The `pad_collate_fn` function is a batching utility for preparing input data for model inference. It aligns input features from multiple sources (tokenizer & feature extractor) by padding values appropriately.

**Key Implementation Details:**

*   Verifies tokenizer and feature extractor configurations.
*   Checks for consistency in padding sides between tokenizer and feature extractor.
*   Applies padding to consistent side of inputs based on `padding_side` configuration.

**Notable Dependencies and Requirements:**

*   `tokenizer`: used for text input formatting
*   `feature_extractor`: used for image input formatting (optional)
    *   `pad_token_id`
    *   `padding_value`

**External Interface:*

The function returns a transformation object (`inner`) that accepts batches of input data.

*   Example usage: `padded = inner([dataset_example])`, where `dataset_example` is an example batch containing input data for the model.


key: chroma:document
string_value:
CODE:

def infer_framework_load_model(
    model,
    config: AutoConfig,
    model_classes: Optional[Dict[str, Tuple[type]]] = None,
    task: Optional[str] = None,
    framework: Optional[str] = None,
    **model_kwargs,
):
    """
    Select framework (TensorFlow or PyTorch) to use from the `model` passed. Returns a tuple (framework, model).

    If `model` is instantiated, this function will just infer the framework from the model class. Otherwise `model` is
    actually a checkpoint name and this method will try to instantiate it using `model_classes`. Since we don't want to
    instantiate the model twice, this model is returned for use by the pipeline.

    If both frameworks are installed and available for `model`, PyTorch is selected.

    Args:
        model (`str`, [`PreTrainedModel`] or [`TFPreTrainedModel]`):
            The model to infer the framework from. If `str`, a checkpoint name. The model to infer the framewrok from.
        config ([`AutoConfig`]):
            The config associated with the model to help using the correct class
        model_classes (dictionary `str` to `type`, *optional*):
            A mapping framework to class.
        task (`str`):
            The task defining which pipeline will be returned.
        model_kwargs:
            Additional dictionary of keyword arguments passed along to the model's `from_pretrained(...,
            **model_kwargs)` function.

    Returns:
        `Tuple`: A tuple framework, model.
    """
    if not is_tf_available() and not is_torch_available():
        raise RuntimeError(
            "At least one of TensorFlow 2.0 or PyTorch should be installed. "
            "To install TensorFlow 2.0, read the instructions at https://www.tensorflow.org/install/ "
            "To install PyTorch, read the instructions at https://pytorch.org/."
        )
    if isinstance(model, str):
        model_kwargs["_from_pipeline"] = task
        class_tuple = ()
        look_pt = is_torch_available() and framework in {"pt", None}
        look_tf = is_tf_available() and framework in {"tf", None}
        if model_classes:
            if look_pt:
                class_tuple = class_tuple + model_classes.get("pt", (AutoModel,))
            if look_tf:
                class_tuple = class_tuple + model_classes.get("tf", (TFAutoModel,))
        if config.architectures:
            classes = []
            for architecture in config.architectures:
                transformers_module = importlib.import_module("transformers")
                if look_pt:
                    _class = getattr(transformers_module, architecture, None)
                    if _class is not None:
                        classes.append(_class)
                if look_tf:
                    _class = getattr(transformers_module, f"TF{architecture}", None)
                    if _class is not None:
                        classes.append(_class)
            class_tuple = class_tuple + tuple(classes)

        if len(class_tuple) == 0:
            raise ValueError(f"Pipeline cannot infer suitable model classes from {model}")

        all_traceback = {}
        for model_class in class_tuple:
            kwargs = model_kwargs.copy()
            if framework == "pt" and model.endswith(".h5"):
                kwargs["from_tf"] = True
                logger.warning(
                    "Model might be a TensorFlow model (ending with `.h5`) but TensorFlow is not available. "
                    "Trying to load the model with PyTorch."
                )
            elif framework == "tf" and model.endswith(".bin"):
                kwargs["from_pt"] = True
                logger.warning(
                    "Model might be a PyTorch model (ending with `.bin`) but PyTorch is not available. "
                    "Trying to load the model with Tensorflow."
                )

            try:
                model = model_class.from_pretrained(model, **kwargs)
                if hasattr(model, "eval"):
                    model = model.eval()
                # Stop loading on the first successful load.
                break
            except (OSError, ValueError):
                all_traceback[model_class.__name__] = traceback.format_exc()
                continue

        if isinstance(model, str):
            error = ""
            for class_name, trace in all_traceback.items():
                error += f"while loading with {class_name}, an error is thrown:\n{trace}\n"
            raise ValueError(
                f"Could not load model {model} with any of the following classes: {class_tuple}. See the original errors:\n\n{error}\n"
            )

    if framework is None:
        framework = infer_framework(model.__class__)


SUMMARY:
**Summary**

Infer Framework Load Model Function
=====================================

### Purpose and Functionality

This function inferences the framework (TensorFlow or PyTorch) from a given model. It returns a tuple containing the framework and loaded model.

### Key Implementation Details

*   Inferences framework based on TensorFlow availability and framework name.
*   Creates a dictionary of models using `model_classes` mapping.
*   Installs necessary modules and checks their availability before loading models.
*   Tries to load multiple classes in sequence, with error handling for each failed attempt.

### Notable Dependencies or Requirements

*   Torch library (`transformers`)
*   TensorFlow library
*   `AutoConfig`
*   `TFPreTrainedModel`

### Return Value

A tuple containing the inferred framework and loaded model.
```python
(framework: str, model)
```


key: chroma:document
string_value:
CODE:

def infer_framework_from_model(
    model,
    model_classes: Optional[Dict[str, Tuple[type]]] = None,
    task: Optional[str] = None,
    framework: Optional[str] = None,
    **model_kwargs,
):
    """
    Select framework (TensorFlow or PyTorch) to use from the `model` passed. Returns a tuple (framework, model).

    If `model` is instantiated, this function will just infer the framework from the model class. Otherwise `model` is
    actually a checkpoint name and this method will try to instantiate it using `model_classes`. Since we don't want to
    instantiate the model twice, this model is returned for use by the pipeline.

    If both frameworks are installed and available for `model`, PyTorch is selected.

    Args:
        model (`str`, [`PreTrainedModel`] or [`TFPreTrainedModel]`):
            The model to infer the framework from. If `str`, a checkpoint name. The model to infer the framewrok from.
        model_classes (dictionary `str` to `type`, *optional*):
            A mapping framework to class.
        task (`str`):
            The task defining which pipeline will be returned.
        model_kwargs:
            Additional dictionary of keyword arguments passed along to the model's `from_pretrained(...,
            **model_kwargs)` function.

    Returns:
        `Tuple`: A tuple framework, model.
    """
    if isinstance(model, str):
        config = AutoConfig.from_pretrained(model, _from_pipeline=task, **model_kwargs)
    else:
        config = model.config
    return infer_framework_load_model(
        model, config, model_classes=model_classes, _from_pipeline=task, task=task, framework=framework, **model_kwargs


SUMMARY:
**Function Summary: `infer_framework_from_model`**

*Purpose*: Selects the framework (TensorFlow or PyTorch) to use with a given model based on its class and configuration.

**Key Implementation Details:**

* Infers framework from model class if model is an instance
* Tries to instantiate model using `model_classes` dictionary if model is a string checkpoint name
* Priority given to PyTorch over TensorFlow when both are installed

**Method Signature:** `infer_framework_from_model(model, model_classes, task, framework, **model_kwargs) -> Tuple[framework, model]`

**Notable Dependencies and Requirements:**

* `AutoConfig.from_pretrained` for loading configuration
* `model.classes` for accessing model configuration
* `model_classes` dictionary for mapping frameworks to classes
* `TensorFlow` and `PyTorch` installations determined at runtime


key: chroma:document
string_value:
CODE:

def get_framework(model, revision: Optional[str] = None):
    """
    Select framework (TensorFlow or PyTorch) to use.

    Args:
        model (`str`, [`PreTrainedModel`] or [`TFPreTrainedModel]`):
            If both frameworks are installed, picks the one corresponding to the model passed (either a model class or
            the model name). If no specific model is provided, defaults to using PyTorch.
    """
    warnings.warn(
        "`get_framework` is deprecated and will be removed in v5, use `infer_framework_from_model` instead.",
        FutureWarning,
    )
    if not is_tf_available() and not is_torch_available():
        raise RuntimeError(
            "At least one of TensorFlow 2.0 or PyTorch should be installed. "
            "To install TensorFlow 2.0, read the instructions at https://www.tensorflow.org/install/ "
            "To install PyTorch, read the instructions at https://pytorch.org/."
        )
    if isinstance(model, str):
        if is_torch_available() and not is_tf_available():
            model = AutoModel.from_pretrained(model, revision=revision)
        elif is_tf_available() and not is_torch_available():
            model = TFAutoModel.from_pretrained(model, revision=revision)
        else:
            try:
                model = AutoModel.from_pretrained(model, revision=revision)
            except OSError:
                model = TFAutoModel.from_pretrained(model, revision=revision)

    framework = infer_framework(model.__class__)


SUMMARY:
**Function Summary: `get_framework`**

### Purpose and Functionality

The `get_framework` function determines the underlying framework (TensorFlow or PyTorch) to use for a given model. It takes a `model` argument, which can be either a string representing the model name or an instance of `PreTrainedModel`/`TFPreTrainedModel`.

### Key Implementation Details

* Uses the `infer_framework_from_model` function to determine the framework based on the model's class.
* Handles different cases for framework availability and assigns the corresponding `AutoModel`/`TFAutoModel` instance.

### Notable Dependencies or Requirements

* TensorFlow 2.0 or PyTorch must be installed.
* In cases where a specific model is not provided, the function defaults to using PyTorch.

### Example Parameters

* `model`: string (model name) or `PreTrainedModel`/`TFPreTrainedModel` instance.

### Return Value
The returned framework instance corresponds to the underlying framework used for the given model.


key: chroma:document
string_value:
CODE:

def get_default_model_and_revision(
    targeted_task: Dict, framework: Optional[str], task_options: Optional[Any]
) -> Union[str, Tuple[str, str]]:
    """
    Select a default model to use for a given task. Defaults to pytorch if ambiguous.

    Args:
        targeted_task (`Dict`):
           Dictionary representing the given task, that should contain default models

        framework (`str`, None)
           "pt", "tf" or None, representing a specific framework if it was specified, or None if we don't know yet.

        task_options (`Any`, None)
           Any further value required by the task to get fully specified, for instance (SRC, TGT) languages for
           translation task.

    Returns

        `str` The model string representing the default model for this pipeline
    """
    if is_torch_available() and not is_tf_available():
        framework = "pt"
    elif is_tf_available() and not is_torch_available():
        framework = "tf"

    defaults = targeted_task["default"]
    if task_options:
        if task_options not in defaults:
            raise ValueError(f"The task does not provide any default models for options {task_options}")
        default_models = defaults[task_options]["model"]
    elif "model" in defaults:
        default_models = targeted_task["default"]["model"]
    else:
        # XXX This error message needs to be updated to be more generic if more tasks are going to become
        # parametrized
        raise ValueError('The task defaults can\'t be correctly selected. You probably meant "translation_XX_to_YY"')

    if framework is None:
        framework = "pt"



SUMMARY:
**Function Summary: `get_default_model_and_revision`**

**Purpose and Functionality:**
This function selects a default model for a given task based on the provided framework and task options. It returns the model string representing the default model for this pipeline.

*   If no explicit framework is specified, it defaults to PyTorch.
*   It checks if specific task options are provided and uses the corresponding default models.
*   It raises an error if the framework or task options are not supported.

**Key Implementation Details:**

*   Uses conditional statements to handle different frameworks (PyTorch/TensorFlow) based on availability.
*   Utilizes a dictionary (`defaults`) to store default models for specific task options or translations.

**Notable Dependencies and Requirements:**

*   The function uses the following external dependencies:
    *   `is_torch_available()` and `is_tf_available()` functions
    *   `Optional` and `Union` types from the standard library
    *   `Any` type as an argument for some function parameters

There are no inherited or implemented interfaces mentioned in this code chunk.


key: chroma:document
string_value:
CODE:

def load_assistant_model(
    model: "PreTrainedModel",
    assistant_model: Optional[Union[str, "PreTrainedModel"]],
    assistant_tokenizer: Optional[PreTrainedTokenizer],
) -> Tuple[Optional["PreTrainedModel"], Optional[PreTrainedTokenizer]]:
    """
    Prepares the assistant model and the assistant tokenizer for a pipeline whose model that can call `generate`.

    Args:
        model ([`PreTrainedModel`]):
            The main model that will be used by the pipeline to make predictions.
        assistant_model (`str` or [`PreTrainedModel`], *optional*):
            The assistant model that will be used by the pipeline to make predictions.
        assistant_tokenizer ([`PreTrainedTokenizer`], *optional*):
            The assistant tokenizer that will be used by the pipeline to encode data for the model.

    Returns:
        Tuple: The loaded assistant model and (optionally) the loaded tokenizer.
    """
    if not model.can_generate() or assistant_model is None:
        return None, None

    if not isinstance(model, PreTrainedModel):
        raise ValueError(
            "Assisted generation, triggered by the `assistant_model` argument, is only available for "
            "`PreTrainedModel` model instances. For instance, TF or JAX models are not supported."
        )

    # If the model is passed as a string, load the model and the corresponding tokenizer
    if isinstance(assistant_model, str):
        assistant_config = AutoConfig.from_pretrained(assistant_model)
        _, loaded_assistant_model = infer_framework_load_model(assistant_model, config=assistant_config)
        loaded_assistant_model = loaded_assistant_model.to(device=model.device, dtype=model.dtype)
        loaded_assistant_tokenizer = AutoTokenizer.from_pretrained(assistant_model)
    else:
        loaded_assistant_model = assistant_model
        loaded_assistant_tokenizer = assistant_tokenizer

    # Finally, let's check the tokenizers: if the two models have different tokenizers, we need to keep the assistant
    # tokenizer
    same_vocab_size = model.config.vocab_size == loaded_assistant_model.config.vocab_size
    same_special_tokens = all(
        getattr(model.config, token) == getattr(loaded_assistant_model.config, token)
        for token in ("eos_token_id", "pad_token_id", "bos_token_id")
    )
    if same_vocab_size and same_special_tokens:
        loaded_assistant_tokenizer = None
    elif loaded_assistant_tokenizer is None:
        raise ValueError(
            "The assistant model has a different tokenizer than the main model. You should pass the assistant "
            "tokenizer."
        )



SUMMARY:
**Function Summary: `load_assistant_model`**

Prepares an assistant model and tokenizer for a pipeline that can perform generation.

```markdown
### Purpose and Functionality
The main purpose of this function is to load the assistant model (`assistant_model`) along with its tokenizer (`assistant_tokenizer`) if applicable. It is essential for pipeline operations that involve assisted generation, which requires both model instances to have identical vocab sizes and token configurations.
### Key Implementation Details

- The function first checks whether the `assistant_model` can generate and if it is a valid `PreTrainedModel`.
- If the `assistant_model` argument is passed as a string, it loads the corresponding model and tokenizer instances using pre-trained AutoConfigs from Hugging Face's library.

### Notable Dependencies and Requirements
The function relies on the following:
- `AutoConfig`
- `InferFrameworkLoadModel`
- `PreTrainedTokenizer` (`assistant_tokenizer`)
- `PreTrainedModel` (`model`, `assistant_model`)
- The user-provided or inferred device type for model operations.

### Return Value
A tuple containing the loaded `assistant_model` and optionally, the loaded `assistant_tokenizer`. If both are the same or if no tokenization is required due to matching vocab size, the `assistant_tokenizer` will be set to None.


key: chroma:document
string_value:
CODE:

class PipelineException(Exception):
    """
    Raised by a [`Pipeline`] when handling __call__.

    Args:
        task (`str`): The task of the pipeline.
        model (`str`): The model used by the pipeline.
        reason (`str`): The error message to display.
    """

    def __init__(self, task: str, model: str, reason: str):
        super().__init__(reason)

        self.task = task


SUMMARY:
**PipelineException Class Summary**

The `PipelineException` class is a custom exception raised by the Pipeline when handling the `__call__` method. It provides information about the task, model used, and error reason.

**Key Implementation Details:**

- Custom exception type.
- Initializes with `task`, `model`, and `reason`.
  - `super().__init__(reason)` sets the error message.
  
**Notable Dependencies or Requirements:**

- [`Pipeline` class] (related class).
- No external dependencies.

**Primary Purpose:**
Raise a custom exception with relevant task, model, and error information when handling the Pipeline's `__call__` method.


key: chroma:document
string_value:
CODE:

class ArgumentHandler(ABC):
    """
    Base interface for handling arguments for each [`~pipelines.Pipeline`].
    """

    @abstractmethod
    def __call__(self, *args, **kwargs):


SUMMARY:
**ArgumentHandler Class Summary**

The `ArgumentHandler` class serves as a base interface for handling arguments specific to each pipeline in a workflow. 

**Key Functionality:**
- Provides a single, abstract constructor (`__call__`) that expects variable number of positional and keyword arguments.

**Notable Implementation Details:**
- Enforces the use of a concrete implementation through the `NotImplementedError` exception, indicating that subclasses must provide an actual implementation for handling arguments.
 
**Dependencies and Requirements:**
- No explicit dependencies are mentioned. However, it is implicitly dependent on the `pipelines.Pipeline` class, referencing its scope.


key: chroma:document
string_value:
CODE:

class PipelineDataFormat:
    """
    Base class for all the pipeline supported data format both for reading and writing. Supported data formats
    currently includes:

    - JSON
    - CSV
    - stdin/stdout (pipe)

    `PipelineDataFormat` also includes some utilities to work with multi-columns like mapping from datasets columns to
    pipelines keyword arguments through the `dataset_kwarg_1=dataset_column_1` format.

    Args:
        output_path (`str`): Where to save the outgoing data.
        input_path (`str`): Where to look for the input data.
        column (`str`): The column to read.
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether or not to overwrite the `output_path`.
    """

    SUPPORTED_FORMATS = ["json", "csv", "pipe"]

    def __init__(
        self,
        output_path: Optional[str],
        input_path: Optional[str],
        column: Optional[str],
        overwrite: bool = False,
    ):
        self.output_path = output_path
        self.input_path = input_path
        self.column = column.split(",") if column is not None else [""]
        self.is_multi_columns = len(self.column) > 1

        if self.is_multi_columns:
            self.column = [tuple(c.split("=")) if "=" in c else (c, c) for c in self.column]

        if output_path is not None and not overwrite:
            if exists(abspath(self.output_path)):
                raise OSError(f"{self.output_path} already exists on disk")

        if input_path is not None:
            if not exists(abspath(self.input_path)):
                raise OSError(f"{self.input_path} doesnt exist on disk")

    @abstractmethod
    def __iter__(self):
        raise NotImplementedError()

    @abstractmethod
    def save(self, data: Union[dict, List[dict]]):
        """
        Save the provided data object with the representation for the current [`~pipelines.PipelineDataFormat`].

        Args:
            data (`dict` or list of `dict`): The data to store.
        """
        raise NotImplementedError()

    def save_binary(self, data: Union[dict, List[dict]]) -> str:
        """
        Save the provided data object as a pickle-formatted binary data on the disk.

        Args:
            data (`dict` or list of `dict`): The data to store.

        Returns:
            `str`: Path where the data has been saved.
        """
        path, _ = os.path.splitext(self.output_path)
        binary_path = os.path.extsep.join((path, "pickle"))

        with open(binary_path, "wb+") as f_output:
            pickle.dump(data, f_output)

        return binary_path

    @staticmethod
    def from_str(
        format: str,
        output_path: Optional[str],
        input_path: Optional[str],
        column: Optional[str],
        overwrite=False,
    ) -> "PipelineDataFormat":
        """
        Creates an instance of the right subclass of [`~pipelines.PipelineDataFormat`] depending on `format`.

        Args:
            format (`str`):
                The format of the desired pipeline. Acceptable values are `"json"`, `"csv"` or `"pipe"`.
            output_path (`str`, *optional*):
                Where to save the outgoing data.
            input_path (`str`, *optional*):
                Where to look for the input data.
            column (`str`, *optional*):
                The column to read.
            overwrite (`bool`, *optional*, defaults to `False`):
                Whether or not to overwrite the `output_path`.

        Returns:
            [`~pipelines.PipelineDataFormat`]: The proper data format.
        """
        if format == "json":
            return JsonPipelineDataFormat(output_path, input_path, column, overwrite=overwrite)
        elif format == "csv":
            return CsvPipelineDataFormat(output_path, input_path, column, overwrite=overwrite)
        elif format == "pipe":
            return PipedPipelineDataFormat(output_path, input_path, column, overwrite=overwrite)
        else:


SUMMARY:
**Summary of `PipelineDataFormat` Class**

The `PipelineDataFormat` class is a base class for various pipeline data formats, including JSON, CSV, and pipe. It provides utilities for working with multi-columns and supports reading/writing data to disk.

### Main Purpose and Functionality

- **Read/Write Data**: Supports reading from and writing to various data formats.
- **Multi-column Handling**: Provides functionality for mapping dataset columns to pipeline keyword arguments.

### Key Implementation Details

*   **`__init__` method**:
    *   Initializes the class with output, input paths, column names, and overwrite flag.
    *   Checks if the output path exists and raises an error if it does not exist or needs to be overwritten.
*   **`save` interface**: Defines a method for saving data to disk, which is implemented by subclasses (JSON or CSV format).
*   **`save_binary` method**: Saves data in binary pickle format on disk.
*   **`from_str` static method**: Creates an instance of the correct subclass based on the specified `format`.

### Notable Dependencies and Requirements

-   No external dependencies are listed; however, it likely requires `os`, `pickle`, and `json` libraries.

Note that there's no inheritance or implemented interfaces explicitly mentioned in this class summary. However, each formatted pipeline (`PipelineDataFormat` subclass) inherits from the base format.


key: chroma:document
string_value:
CODE:

class CsvPipelineDataFormat(PipelineDataFormat):
    """
    Support for pipelines using CSV data format.

    Args:
        output_path (`str`): Where to save the outgoing data.
        input_path (`str`): Where to look for the input data.
        column (`str`): The column to read.
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether or not to overwrite the `output_path`.
    """

    def __init__(
        self,
        output_path: Optional[str],
        input_path: Optional[str],
        column: Optional[str],
        overwrite=False,
    ):
        super().__init__(output_path, input_path, column, overwrite=overwrite)

    def __iter__(self):
        with open(self.input_path, "r") as f:
            reader = csv.DictReader(f)
            for row in reader:
                if self.is_multi_columns:
                    yield {k: row[c] for k, c in self.column}
                else:
                    yield row[self.column[0]]

    def save(self, data: List[dict]):
        """
        Save the provided data object with the representation for the current [`~pipelines.PipelineDataFormat`].

        Args:
            data (`List[dict]`): The data to store.
        """
        with open(self.output_path, "w") as f:
            if len(data) > 0:
                writer = csv.DictWriter(f, list(data[0].keys()))
                writer.writeheader()


SUMMARY:
**CsvPipelineDataFormat Class Summary**

The `CsvPipelineDataFormat` class provides support for pipelines using CSV data format. Its main purpose is to read and write CSV data.

**Primary Purpose:**
To load and save CSV data from `input_path` to `output_path` based on a specified column.

**Key Implementation Details:**

* The `__iter__` method iterates over the CSV rows, yielding a dictionary representation (dict) of each row.
* The `save` method writes data to a CSV file at `output_path`.

**Method Signatures and Interfaces:**

* `__init__(output_path: Optional[str], input_path: Optional[str], column: Optional[str], overwrite=False)`
* `__iter__()`
* `save(data: List[dict])`

**Notable Dependencies or Requirements:**

* `csv.DictReader` and `csv.DictWriter` for reading and writing CSV data, respectively.
* The `pipelines.PipelineDataFormat` interface is inherited.

**External Dependencies:**

* The `csv` module (part of the Python Standard Library)
* Assumed dependency on other pipelines classes in the `pipelines` package.


key: chroma:document
string_value:
CODE:

class JsonPipelineDataFormat(PipelineDataFormat):
    """
    Support for pipelines using JSON file format.

    Args:
        output_path (`str`): Where to save the outgoing data.
        input_path (`str`): Where to look for the input data.
        column (`str`): The column to read.
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether or not to overwrite the `output_path`.
    """

    def __init__(
        self,
        output_path: Optional[str],
        input_path: Optional[str],
        column: Optional[str],
        overwrite=False,
    ):
        super().__init__(output_path, input_path, column, overwrite=overwrite)

        with open(input_path, "r") as f:
            self._entries = json.load(f)

    def __iter__(self):
        for entry in self._entries:
            if self.is_multi_columns:
                yield {k: entry[c] for k, c in self.column}
            else:
                yield entry[self.column[0]]

    def save(self, data: dict):
        """
        Save the provided data object in a json file.

        Args:
            data (`dict`): The data to store.
        """
        with open(self.output_path, "w") as f:


SUMMARY:
**JsonPipelineDataFormat Class Summary**

### Purpose and Functionality

The `JsonPipelineDataFormat` class is designed for Pipelines using JSON file format. It supports reading input data from a JSON file, processing the data through various columns (if applicable), and writing processed data to an output JSON file.

### Key Implementation Details

*   Reads input data from a specified JSON file and stores it in `_entries`.
*   Implements a custom iterator (`__iter__`) that yields either individual entries or columns depending on `is_multi_columns` setting.
*   Saves the provided `data` object in a formatted JSON file using `json.dump`.

### Notable Dependencies and Requirements

*   **Python Standard Library:** `json`, `io`
*   **Optional Packages/(Classes):** `PipelineDataFormat`

### Key Method Signatures and Interfaces

*   `__init__(output_path: str, input_path: str, column: str, overwrite: bool = False)`
*   `save(data: dict)`
*   `is_multi_columns.setter` (implies `is_multi_columns` is a property)


key: chroma:document
string_value:
CODE:

class PipedPipelineDataFormat(PipelineDataFormat):
    """
    Read data from piped input to the python process. For multi columns data, columns should separated by \t

    If columns are provided, then the output will be a dictionary with {column_x: value_x}

    Args:
        output_path (`str`): Where to save the outgoing data.
        input_path (`str`): Where to look for the input data.
        column (`str`): The column to read.
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether or not to overwrite the `output_path`.
    """

    def __iter__(self):
        for line in sys.stdin:
            # Split for multi-columns
            if "\t" in line:
                line = line.split("\t")
                if self.column:
                    # Dictionary to map arguments
                    yield {kwargs: l for (kwargs, _), l in zip(self.column, line)}
                else:
                    yield tuple(line)

            # No dictionary to map arguments
            else:
                yield line

    def save(self, data: dict):
        """
        Print the data.

        Args:
            data (`dict`): The data to store.
        """
        print(data)

    def save_binary(self, data: Union[dict, List[dict]]) -> str:
        if self.output_path is None:
            raise KeyError(
                "When using piped input on pipeline outputting large object requires an output file path. "
                "Please provide such output path through --output argument."
            )



SUMMARY:
**PipedPipelineDataFormat Class Summary**

### Purpose and Functionality
Implements data pipeline for piped input, producing dictionaries or tuples based on specified columns.

### Implementation Details

*   **`__iter__` method**: Iterates over system stdin, splitting lines into multiple columns (if separated by `\t`) and yielding dictionaries with column names as keys.
*   **`save` method**: Prints data as a dictionary.
*   **`save_binary` method**: Saves binary data to output path (requires `output_path` argument). Calls superclass method if not provided.

### Dependencies and Requirements

*   Uses `sys.stdin` for piped input
*   Requires `PipelineDataFormat` superclass (not shown in snippet)
*   Assumes Python's built-in `print` function and dictionary data type.
*   No external dependencies noted.


key: chroma:document
string_value:
CODE:

class _ScikitCompat(ABC):
    """
    Interface layer for the Scikit and Keras compatibility.
    """

    @abstractmethod
    def transform(self, X):
        raise NotImplementedError()

    @abstractmethod
    def predict(self, X):


SUMMARY:
**Summary:**
```python
class _ScikitCompat(ABC):
    """
    Interface layer for Scikit/Keras compatibility.
    
    Provides abstract methods for transform and predict operations.
    """

    @abstractmethod
    def transform(self, X):
        # Not implemented; raised on call
    
    @abstractmethod
    def predict(self, X):
        # Not implemented; raised on call
```

**Analysis:**

* The `_ScikitCompat` class serves as an interface for compatibility between Scikit and Keras libraries.
* It has two abstract methods:
	+ `transform(X)`: not implemented, raises `NotImplementedError`.
	+ `predict(X)`: not implemented, raises `NotImplementedError`.

**Key Points:**

* Defines the interface layer for compatibility between Scikit and Keras libraries.
* Abstracts transform and predict operations with empty implementations.
* Notable dependency: None (builtin Python/standard library).

Note: The summary is concise and focused on the main purpose, key implementation details, and notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

def build_pipeline_init_args(
    has_tokenizer: bool = False,
    has_feature_extractor: bool = False,
    has_image_processor: bool = False,
    has_processor: bool = False,
    supports_binary_output: bool = True,
) -> str:
    docstring = r"""
    Arguments:
        model ([`PreTrainedModel`] or [`TFPreTrainedModel`]):
            The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from
            [`PreTrainedModel`] for PyTorch and [`TFPreTrainedModel`] for TensorFlow."""
    if has_tokenizer:
        docstring += r"""
        tokenizer ([`PreTrainedTokenizer`]):
            The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from
            [`PreTrainedTokenizer`]."""
    if has_feature_extractor:
        docstring += r"""
        feature_extractor ([`SequenceFeatureExtractor`]):
            The feature extractor that will be used by the pipeline to encode data for the model. This object inherits from
            [`SequenceFeatureExtractor`]."""
    if has_image_processor:
        docstring += r"""
        image_processor ([`BaseImageProcessor`]):
            The image processor that will be used by the pipeline to encode data for the model. This object inherits from
            [`BaseImageProcessor`]."""
    if has_processor:
        docstring += r"""
        processor ([`ProcessorMixin`]):
            The processor that will be used by the pipeline to encode data for the model. This object inherits from
            [`ProcessorMixin`]. Processor is a composite object that might contain `tokenizer`, `feature_extractor`, and
            `image_processor`."""
    docstring += r"""
        modelcard (`str` or [`ModelCard`], *optional*):
            Model card attributed to the model for this pipeline.
        framework (`str`, *optional*):
            The framework to use, either `"pt"` for PyTorch or `"tf"` for TensorFlow. The specified framework must be
            installed.

            If no framework is specified, will default to the one currently installed. If no framework is specified and
            both frameworks are installed, will default to the framework of the `model`, or to PyTorch if no model is
            provided.
        task (`str`, defaults to `""`):
            A task-identifier for the pipeline.
        num_workers (`int`, *optional*, defaults to 8):
            When the pipeline will use *DataLoader* (when passing a dataset, on GPU for a Pytorch model), the number of
            workers to be used.
        batch_size (`int`, *optional*, defaults to 1):
            When the pipeline will use *DataLoader* (when passing a dataset, on GPU for a Pytorch model), the size of
            the batch to use, for inference this is not always beneficial, please read [Batching with
            pipelines](https://huggingface.co/transformers/main_classes/pipelines.html#pipeline-batching) .
        args_parser ([`~pipelines.ArgumentHandler`], *optional*):
            Reference to the object in charge of parsing supplied pipeline parameters.
        device (`int`, *optional*, defaults to -1):
            Device ordinal for CPU/GPU supports. Setting this to -1 will leverage CPU, a positive will run the model on
            the associated CUDA device id. You can pass native `torch.device` or a `str` too
        torch_dtype (`str` or `torch.dtype`, *optional*):
            Sent directly as `model_kwargs` (just a simpler shortcut) to use the available precision for this model
            (`torch.float16`, `torch.bfloat16`, ... or `"auto"`)"""
    if supports_binary_output:
        docstring += r"""
        binary_output (`bool`, *optional*, defaults to `False`):
            Flag indicating if the output the pipeline should happen in a serialized format (i.e., pickle) or as
            the raw output data e.g. text."""


SUMMARY:
**Function Summary:**

`build_pipeline_init_args`

Builds and returns an initial argument string for a pipeline, incorporating model, tokenizer, feature extractor, image processor, and other parameters.

**Key Implementation Details:**

* Accepts boolean flags to determine the presence of various components.
* Incorporates documentation strings with detailed descriptions of each parameter.
* Uses optional parameters (e.g., `modelcard`, `framework`) for fine-tuning pipeline configuration.

**Notable Dependencies/Requirements:**

* `PreTrainedModel` and related classes (`TFPreTrainedModel`, `PreTrainedTokenizer`, etc.)
* `SequenceFeatureExtractor`
* `BaseImageProcessor`
* `ProcessorMixin`
* Optional dependencies:
	+ PyTorch (`torch`)
	+ TensorFlow (`tf`)
	+ Other frameworks (e.g., `"pt"` or `"tf"`)


key: chroma:document
string_value:
CODE:

class ChunkPipeline(Pipeline):
    def run_single(self, inputs, preprocess_params, forward_params, postprocess_params):
        all_outputs = []
        for model_inputs in self.preprocess(inputs, **preprocess_params):
            model_outputs = self.forward(model_inputs, **forward_params)
            all_outputs.append(model_outputs)
        outputs = self.postprocess(all_outputs, **postprocess_params)
        return outputs

    def get_iterator(
        self, inputs, num_workers: int, batch_size: int, preprocess_params, forward_params, postprocess_params
    ):
        if "TOKENIZERS_PARALLELISM" not in os.environ:
            logger.info("Disabling tokenizer parallelism, we're using DataLoader multithreading already")
            os.environ["TOKENIZERS_PARALLELISM"] = "false"
        if num_workers > 1:
            logger.warning(
                "For ChunkPipeline using num_workers>0 is likely to result in errors since everything is iterable,"
                " setting `num_workers=1` to guarantee correctness."
            )
            num_workers = 1
        dataset = PipelineChunkIterator(inputs, self.preprocess, preprocess_params)

        # TODO hack by collating feature_extractor and image_processor
        feature_extractor = self.feature_extractor if self.feature_extractor is not None else self.image_processor
        collate_fn = no_collate_fn if batch_size == 1 else pad_collate_fn(self.tokenizer, feature_extractor)
        dataloader = DataLoader(dataset, num_workers=num_workers, batch_size=batch_size, collate_fn=collate_fn)
        model_iterator = PipelinePackIterator(dataloader, self.forward, forward_params, loader_batch_size=batch_size)
        final_iterator = PipelineIterator(model_iterator, self.postprocess, postprocess_params)


SUMMARY:
**Summary:**

The `ChunkPipeline` class extends the `Pipeline` class and provides a data processing pipeline for batches of input data. The primary purpose is to preprocess, forward process, and postprocess individual models' inputs.

**Main Functionality:**

1.  **run_single:** Runs a single iteration through the pipeline, applying preprocessing, forward processing, and postprocessing steps.
2.  **get_iterator:** Returns an iterator for the pipeline, collecting batches of input data with preprocessing, forward processing, and postprocessing applied.

**Key Implementation Details:**

*   Uses `PipelineChunkIterator` to iterate over the dataset, featuring a custom collate function that handles single-sample batches using `no_collate_fn`.
*   Employs `DataLoader` from PyTorch for efficient batch loading.
*   Leverages `PipelinePackIterator` and `PipelineIterator` for further processing and postprocessing of model inputs.

**Notable Dependencies and Requirements:**

*   ` Pipeline, PipelineChunkIterator, PipelinePackIterator, PipelineIterator`
*   `os.environ['TOKENIZERS_PARALLELISM']` is set to "false" when not using parallelism.
*   External dependents may include:
    *   `no_collate_fn`
    *   `pad_collate_fn`
    *   `tokenizer`
    *   `feature_extractor` or `image_processor`

(Note: No external dependencies have been explicitly mentioned in the code snippet, but it's assumed these might be imported from other modules.)


key: chroma:document
string_value:
CODE:

class PipelineRegistry:
    def __init__(self, supported_tasks: Dict[str, Any], task_aliases: Dict[str, str]) -> None:
        self.supported_tasks = supported_tasks
        self.task_aliases = task_aliases

    def get_supported_tasks(self) -> List[str]:
        supported_task = list(self.supported_tasks.keys()) + list(self.task_aliases.keys())
        supported_task.sort()
        return supported_task

    def check_task(self, task: str) -> Tuple[str, Dict, Any]:
        if task in self.task_aliases:
            task = self.task_aliases[task]
        if task in self.supported_tasks:
            targeted_task = self.supported_tasks[task]
            return task, targeted_task, None

        if task.startswith("translation"):
            tokens = task.split("_")
            if len(tokens) == 4 and tokens[0] == "translation" and tokens[2] == "to":
                targeted_task = self.supported_tasks["translation"]
                task = "translation"
                return task, targeted_task, (tokens[1], tokens[3])
            raise KeyError(f"Invalid translation task {task}, use 'translation_XX_to_YY' format")

        raise KeyError(
            f"Unknown task {task}, available tasks are {self.get_supported_tasks() + ['translation_XX_to_YY']}"
        )

    def register_pipeline(
        self,
        task: str,
        pipeline_class: type,
        pt_model: Optional[Union[type, Tuple[type]]] = None,
        tf_model: Optional[Union[type, Tuple[type]]] = None,
        default: Optional[Dict] = None,
        type: Optional[str] = None,
    ) -> None:
        if task in self.supported_tasks:
            logger.warning(f"{task} is already registered. Overwriting pipeline for task {task}...")

        if pt_model is None:
            pt_model = ()
        elif not isinstance(pt_model, tuple):
            pt_model = (pt_model,)

        if tf_model is None:
            tf_model = ()
        elif not isinstance(tf_model, tuple):
            tf_model = (tf_model,)

        task_impl = {"impl": pipeline_class, "pt": pt_model, "tf": tf_model}

        if default is not None:
            if "model" not in default and ("pt" in default or "tf" in default):
                default = {"model": default}
            task_impl["default"] = default

        if type is not None:
            task_impl["type"] = type

        self.supported_tasks[task] = task_impl
        pipeline_class._registered_impl = {task: task_impl}

    def to_dict(self):


SUMMARY:
**PipelineRegistry Class Summary**

### Overview

The `PipelineRegistry` class manages a dictionary of supported tasks, their aliases, and corresponding pipeline implementations. It provides methods for registering pipelines, checking task validity, and retrieving the registry as a dictionary.

### Key Features

*   Registers pipelines with optional PT (Python Torch) and TF (TensorFlow) models
*   Supports task aliases and automatic translation task detection
*   Enforces unique task registration and raises errors for invalid tasks
*   Exposes the supported tasks as a digestible list

### Notable Methods

*   `__init__(supported_tasks, task_aliases)`: Initializes the registry with support tasks and task aliases.
*   `get_supported_tasks()`: Returns an alphabetically sorted list of all supported tasks, including translation variants.
*   `check_task(task)`: Validates a given task string and returns its corresponding implementation details or raises a KeyError if invalid.
*   `register_pipeline(task, pipeline_class, ... )`: Registers a new pipeline for a specific task, handling model types, defaults, and optional parameters.

### Dependencies

*   No external dependencies (only utilizes Python's built-in modules)
*   Utilizes the following classes/functions:
    *   `Dict`: Dictionary data structure
    *   `List`: List data structure
    *   `Tuple`: Tuple data structure
    *   `Optional`: Optional type hint
    *   `logger.warning()`: Logger warning function

### Interfaces Implemented

*   The registry implements an internal dictionary interface (`self.supported_tasks`) for storing task registrations.


key: chroma:document
string_value:
CODE:

class ReturnType(enum.Enum):
    TENSORS = 0
    NEW_TEXT = 1


SUMMARY:
**Summary:**

This code defines an enumeration (enum) called `ReturnType` with three possible values:

- `TENSORS`: represents tensors
- `NEW_TEXT`: represents new text
- `FULL_TEXT`: represents full text

The primary purpose is to provide a distinct and typed return type for various operations, likely in a data processing or machine learning context.

**Implementation Details:**

* No specific algorithm or design choice is highlighted.
* The method signatures or interfaces are not mentioned as key implementation details.

**Notable Dependencies or Requirements:**

- None explicitly stated. However, this enum is likely used within a class or function that may depend on external libraries or frameworks for tensor operations.


key: chroma:document
string_value:
CODE:

class Chat:
    """This class is intended to just be used internally in this pipeline and not exposed to users. We convert chats
    to this format because the rest of the pipeline code tends to assume that lists of messages are
    actually a batch of samples rather than messages in the same conversation."""

    def __init__(self, messages: Dict, images: Union[str, List[str], "Image.Image", List["Image.Image"]]):
        for message in messages:
            if not ("role" in message and "content" in message):
                raise ValueError("When passing chat dicts as input, each dict must have a 'role' and 'content' key.")
        images = retrieve_images_in_messages(messages, images)

        self.messages = messages


SUMMARY:
**Class Chat Summary**

The `Chat` class represents a conversation in the pipeline. Its primary purpose is to convert chats into a standardized format, where each message has 'role' and 'content' keys.

### Main Purpose and Functionality

* Initializes a chat object with a list of messages and optional images.
* Verifies that each message has required 'role' and 'content' keys.
* Retrieves images from messages using the `retrieve_images_in_messages` function.

### Key Implementation Details

* Method signature: `__init__(messages: Dict, images: Union[str, List[str], "Image.Image", List["Image.Image"]])`.

### Notable Dependencies or Requirements

* No external dependencies listed. However, it assumes the existence of the `retrieve_images_in_messages` function and the `Image` class.

Note: This summary preserves precise technical terms ("Dict", "Union", "List", "Image"), class/function names exactly ("Chat", "__init__"), and states the primary purpose while focusing on key algorithms, patterns, or design choices.


key: chroma:document
string_value:
CODE:

def retrieve_images_in_messages(
    messages: dict, images: Optional[Union[str, List[str], "Image.Image", List["Image.Image"]]]
):
    """
    Retrieve and combine images from the chat and the images passed as input.
    """
    if images is None:
        images = []
    idx_images = 0
    retrieved_images = []
    for message in messages:
        for content in message["content"]:
            if isinstance(content, dict):
                if content.get("type") == "image":
                    for key in ["image", "url", "path", "base64"]:
                        if key in content:
                            retrieved_images.append(content[key])
                            break
                    else:
                        if idx_images < len(images):
                            retrieved_images.append(images[idx_images])
                            idx_images += 1
                        else:
                            raise ValueError(
                                "The number of images in the chat messages should be the same as the number of images passed to the pipeline."
                            )
                # Add support for OpenAI/TGI chat format
                elif content.get("type") == "image_url":
                    if isinstance(content.get("image_url"), dict) and "url" in content["image_url"]:
                        retrieved_images.append(content["image_url"]["url"])
                        # Rewrite content to be in the Transformers chat format
                        content["type"] = "image"
                        content["image"] = content["image_url"]["url"]
                        del content["image_url"]
                    else:
                        raise ValueError(
                            "Wrong format for 'image_url' content type. The content should have an 'image_url' dict with a 'url' key."
                        )

    # The number of images passed should be consistent with the number of images in the chat without an image key
    if idx_images != len(images):
        raise ValueError(
            "The number of images in the chat messages should be the same as the number of images passed to the pipeline."
        )



SUMMARY:
**Summary: retrieve_images_in_messages function**

Purpose:
This function retrieves and combines images from chat messages (`messages`) with additional images (`images`). The primary purpose is to normalize image formats for further processing.

Key Implementation Details:

*   Iterates through each message, content, and key-value pairs.
*   Extracts `image`, `url`, `path`, or `base64` based on `content.type = "image"` or `.type = "image_url"`
*   Handles inconsistent image counts by using the first `images` count as a default
*   Supports OpenAI/TGI chat format by rewriting content
*   Raises errors for invalid formats

Notable Dependencies:
- `messages`: Expected to be a dictionary of messages.
- `images`: Can be any of `str`, `[str]`, `Image.Image`, or `[Image.Image]`.

Method Signatures and Interfaces:

*   None, but it can be adapted to other image-related interfaces.

External Dependencies: 

*   Assume an "Image" class or interface (imported as `Image`) is available, handling image-specific tasks.


key: chroma:document
string_value:
CODE:

class PipelineDataset(Dataset):
    def __init__(self, dataset, process, params):
        self.dataset = dataset
        self.process = process
        self.params = params

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, i):
        item = self.dataset[i]
        processed = self.process(item, **self.params)


SUMMARY:
**Concise Summary**

The `PipelineDataset` class wraps a dataset with data processing functionality. It inherits from Django's `Dataset` and extends it.

**Key Implementations:**

* The class constructor (`__init__`) initializes the dataset, process, and parameters.
* The `__getitem__` method retrieves an item from the dataset, applies the specified processing function (`process`), and returns the processed result.
* The `__len__` method returns the total length of the wrapped dataset.

**Notable Dependencies:**

* Django's `Dataset`
* User-defined `dataset`, `process`, and `params` (not shown in this code snippet)


key: chroma:document
string_value:
CODE:

class PipelineIterator(IterableDataset):
    def __init__(self, loader, infer, params, loader_batch_size=None):
        """
        Roughly equivalent to

        ```
        for item in loader:
            yield infer(item, **params)
        ```

                Arguments:
                    loader (`torch.utils.data.DataLoader` or `Iterable`):
                        The iterator that will be used to apply `infer` on.
                    infer (any function):
                        The function to apply of each element of `loader`.
                    params (`dict`):
                        The parameters passed to `infer` along with every item
                    loader_batch_size (`int`, *optional*):
                        If specified, the items of `loader` are supposed to come as batch, and are loader_batched here
                        making it roughly behave as


        ```
        for items in loader:
            for i in loader_batch_size:
                item = items[i]
                yield infer(item, **params)
        ```"""
        self.loader = loader
        self.infer = infer
        self.params = params
        if loader_batch_size == 1:
            # Let's spare some time by deactivating altogether
            loader_batch_size = None
        self.loader_batch_size = loader_batch_size

        # Internal bookkeeping
        self._loader_batch_index = None
        self._loader_batch_data = None

    def __len__(self):
        return len(self.loader)

    def __iter__(self):
        self.iterator = iter(self.loader)
        return self

    def loader_batch_item(self):
        """
        Return item located at `loader_batch_index` within the current `loader_batch_data`.
        """
        if isinstance(self._loader_batch_data, torch.Tensor):
            # Batch data is simple tensor, just fetch the slice
            result = self._loader_batch_data[self._loader_batch_index].unsqueeze(0)
        else:
            # Batch data is assumed to be BaseModelOutput (or dict)
            loader_batched = {}
            for k, element in self._loader_batch_data.items():
                if isinstance(element, ModelOutput):
                    # Convert ModelOutput to tuple first
                    element = element.to_tuple()
                    if isinstance(element[0], torch.Tensor):
                        loader_batched[k] = tuple(el[self._loader_batch_index].unsqueeze(0) for el in element)
                    elif isinstance(element[0], np.ndarray):
                        loader_batched[k] = tuple(np.expand_dims(el[self._loader_batch_index], 0) for el in element)
                    continue
                if k in {"hidden_states", "past_key_values", "attentions"} and isinstance(element, tuple):
                    # Those are stored as lists of tensors so need specific unbatching.
                    if isinstance(element[0], torch.Tensor):
                        loader_batched[k] = tuple(el[self._loader_batch_index].unsqueeze(0) for el in element)
                    elif isinstance(element[0], np.ndarray):
                        loader_batched[k] = tuple(np.expand_dims(el[self._loader_batch_index], 0) for el in element)
                    continue
                if element is None:
                    # This can happen for optional data that get passed around
                    loader_batched[k] = None
                elif isinstance(element[self._loader_batch_index], torch.Tensor):
                    # Take correct batch data, but make it looked like batch_size=1
                    # For compatibility with other methods within transformers

                    loader_batched[k] = element[self._loader_batch_index].unsqueeze(0)
                elif isinstance(element[self._loader_batch_index], np.ndarray):
                    # Take correct batch data, but make it looked like batch_size=1
                    # For compatibility with other methods within transformers
                    loader_batched[k] = np.expand_dims(element[self._loader_batch_index], 0)
                else:
                    # This is typically a list, so no need to `unsqueeze`.
                    loader_batched[k] = element[self._loader_batch_index]
            # Recreate the element by reusing the original class to make it look
            # batch_size=1
            result = self._loader_batch_data.__class__(loader_batched)
        self._loader_batch_index += 1
        return result

    def __next__(self):
        if self._loader_batch_index is not None and self._loader_batch_index < self.loader_batch_size:
            # We are currently unrolling a batch so we just need to return
            # the current item within a batch
            return self.loader_batch_item()

        # We're out of items within a batch
        item = next(self.iterator)
        processed = self.infer(item, **self.params)
        # We now have a batch of "inferred things".
        if self.loader_batch_size is not None:
            # Try to infer the size of the batch
            if isinstance(processed, torch.Tensor):
                first_tensor = processed
            elif isinstance(processed, tuple):
                first_tensor = processed[0]
            else:
                key = list(processed.keys())[0]
                first_tensor = processed[key]

            if isinstance(first_tensor, list):
                observed_batch_size = len(first_tensor)
            else:
                observed_batch_size = first_tensor.shape[0]
            if 0 < observed_batch_size < self.loader_batch_size:
                # could be last batch so we can't unroll as many
                # elements.
                self.loader_batch_size = observed_batch_size
            # Setting internal index to unwrap the batch
            self._loader_batch_data = processed[0] if isinstance(processed, tuple) else processed
            self._loader_batch_index = 0
            return self.loader_batch_item()
        else:
            # We're not unrolling batches


SUMMARY:
Here's a concise summary of the `PipelineIterator` class:

**Main Purpose and Functionality:**
The `PipelineIterator` is an iterable class that applies a given function (`infer`) to each item in a data loader, with optional batching. It uses internal bookkeeping to manage the iteration process.

**Key Implementation Details:**

* Employs batched iteration of the data loader using `loader_batch_size`.
* Utilizes internal variables `_loader_batch_data` and `_loader_batch_index` to track the current batch.
* Uses `next()` function to retrieve the next item in the iteration, optionally unrolling a batch if `loader_batch_size` is specified.

**Notable Dependencies:**

* `torch.utils.data.DataLoader` or `Iterable`
* `infer` function (any type)
* `params` dictionary

This class appears to be part of a larger transformer framework, possibly related to model inference and data iteration.


key: chroma:document
string_value:
CODE:

class PipelineChunkIterator(PipelineIterator):
    def __init__(self, loader, infer, params, loader_batch_size=None):
        """
        Roughly equivalent to

        ```
        for iterator in loader:
            for item in iterator:
                yield infer(item, **params)
        ```

                Arguments:
                    loader (`torch.utils.data.DataLoader` or `Iterable`):
                        The iterator that will be used to apply `infer` on.
                    infer (any function):
                        The function to apply of each element of `loader`.
                    params (`dict`):
                        The parameters passed to `infer` along with every item
        """
        super().__init__(loader, infer, params)

    def __iter__(self):
        self.iterator = iter(self.loader)
        self.subiterator = None
        return self

    def __next__(self):
        if self.subiterator is None:
            "Subiterator None means we haven't started a `preprocess` iterator. so start it"
            self.subiterator = self.infer(next(self.iterator), **self.params)
        try:
            # Try to return next item
            processed = next(self.subiterator)
        except StopIteration:
            # When a preprocess iterator ends, we can start lookig at the next item
            # ChunkIterator will keep feeding until ALL elements of iterator
            # all have created their subiterator and have been iterating against.
            #
            # Another way to look at it, is we're basically flattening lists of lists
            # into a single list, but with generators
            self.subiterator = self.infer(next(self.iterator), **self.params)
            processed = next(self.subiterator)


SUMMARY:
**PipelineChunkIterator Class Summary**

**Main Purpose and Functionality**
The `PipelineChunkIterator` class is an iterator that applies an inference function `infer` to each element of an input iterator (`loader`) in chunks, utilizing two nested iterators for parallel processing. It feeds the input data through a pipeline where each element is first processed by `infer`, and then yielded as part of the main iterator.

**Key Implementation Details**

* Two nested iterators (`iterator` and `subiterator`) are used to achieve parallel processing.
* The `__next__` method handles iteration over both main and subiterators, switching between them when necessary.
* The `infer` function is applied to each element of the input iterator multiple times in a nested manner.

**Notable Dependencies or Requirements**

* `torch.utils.data.DataLoader` or `Iterable`: the input iterator that will be used to apply `infer` on.
* External dependency: None listed, but `infer` assumes the existence of related classes (e.g., `DataLoader`, `Iterable`) and functions.


key: chroma:document
string_value:
CODE:

class PipelinePackIterator(PipelineIterator):
    """
    Roughly equivalent to

    ```
    packed =  []
    for item in loader:
        packed.append(item)
        if item["is_last"]:
            yield packed
            packed = []
    ```

        but it also handles cases where `item` are batched (meaning it's a dict of Tensor with first dimension > 1. In
        that case it does

    ```
    packed =  []
    for batch in loader:
        # item is batched
        for item in batch:
            packed.append(item)
            if item["is_last"]:
                yield packed
                packed = []
    ```

        Arguments:
            loader (`torch.utils.data.DataLoader` or `Iterable`):
                The iterator that will be used to apply `infer` on.
            infer (any function):
                The function to apply of each element of `loader`.
            params (`dict`):
                The parameters passed to `infer` along with every item
            loader_batch_size (`int`, *optional*):
                If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making
                it roughly behave as


    ```
    for items in loader:
        for i in loader_batch_size:
            item = items[i]
            yield infer(item, **params)
    ```"""

    def __iter__(self):
        self.iterator = iter(self.loader)
        return self

    def __next__(self):
        # Extremely similar to PipelineIterator in its unpacking mechanism
        # BUT, we have an extra required item which is the presence of `is_last`
        # That is because everything is flattened by `PipelineChunkIterator` we
        # need to keep track of how to regroup here in the original `process`
        # boundaries so that `process` and `postprocess` see the same data.

        # This iterator accumulates items (possibly while unbatching) until it
        # its a `is_last` and then just passes it on to the caller.
        is_last = False
        accumulator = []
        if self._loader_batch_index is not None and self._loader_batch_index < self.loader_batch_size:
            while self._loader_batch_index < self.loader_batch_size:
                item = self.loader_batch_item()
                is_last = item.pop("is_last")
                accumulator.append(item)
                if is_last:
                    return accumulator

        while not is_last:
            processed = self.infer(next(self.iterator), **self.params)
            if self.loader_batch_size is not None:
                if isinstance(processed, torch.Tensor):
                    first_tensor = processed
                else:
                    key = list(processed.keys())[0]
                    first_tensor = processed[key]
                if isinstance(first_tensor, list):
                    observed_batch_size = len(first_tensor)
                else:
                    observed_batch_size = first_tensor.shape[0]
                if 0 < observed_batch_size < self.loader_batch_size:
                    # could be last batch so we can't unroll as many
                    # elements.
                    self.loader_batch_size = observed_batch_size
                self._loader_batch_data = processed
                self._loader_batch_index = 0
                while self._loader_batch_index < self.loader_batch_size:
                    item = self.loader_batch_item()
                    is_last = item.pop("is_last")
                    accumulator.append(item)
                    if is_last:
                        return accumulator
            else:
                item = processed
                is_last = item.pop("is_last")
                accumulator.append(item)


SUMMARY:
**Summary**

`PipelinePackIterator` is a custom iterator that extends `PipelineIterator`. It efficiently applies an inference function to batches of items from a data loader, handling both single-item and batched cases. 

**Main Purpose and Functionality:**
- Takes an iterable as input instead of a tensor with dimension 1 (batched tensor).
 
 **Key Implementation Details:**

*   Iterates over the loader, appending each item to an accumulator until a sentinel value ("is_last") is encountered.
*   Unbatching works similarly by iterating over the batch size iterations. If no batches are expected and it encounters a tensor not equal to its single element in the tensor (shape or dict), it adjusts the batch size.
*   Handles both cases of flattened vs batched item handling efficiently.

 **Notable Dependencies/Requirements:**
-   `torch.utils.data.DataLoader`
-   The inference function (`infer`)
-   Optional parameters (`params` and loader_batch_size)
 
**Example usage**: 

```python
# usage example for PipelinePackIterator
loader = torch.utils.data.DataLoader(...)  # data loader as input
infer_func = (...)  # the function to apply on each item of `loader`
chunk_iter = PipelinePackIterator(loader, infer=infer_func)  # create an instance with load and infer arguments

for chunk in chunk_iter:
    print(chunk)  # prints a batched iterator chunk if any else printed single items
    ...    # break loop is you wish (example purposes)
```

 The class implements batching handling to improve efficiency.


key: chroma:document
string_value:
CODE:

class KeyDataset(Dataset):
    def __init__(self, dataset: Dataset, key: str):
        self.dataset = dataset
        self.key = key

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, i):


SUMMARY:
**KeyDataset Class Summary**

* **Main Purpose and Functionality**: A wrapper class around another `Dataset` instance, allowing for data selection via a specific key.

**Implementation Details**:
- Uses the original dataset's index to retrieve value at specified key (``self.dataset[i][self.key]``).

**Notable Dependencies**:
- Inherited from `Dataset`.
- Requires an instance of `Dataset` with associated key during initialization.


key: chroma:document
string_value:
CODE:

class KeyPairDataset(Dataset):
    def __init__(self, dataset: Dataset, key1: str, key2: str):
        self.dataset = dataset
        self.key1 = key1
        self.key2 = key2

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, i):


SUMMARY:
**Class Summary: `KeyPairDataset`**

* **Main Purpose and Functionality**: Custom Dataset class that transforms a base dataset into a `KeyPairDataset` format, where each data point is paired with two distinct keys (`key1` and `key2`).
* **Implementation Details**:
	+ Exposes the `_len()` method to return the length of the underlying dataset.
	+ Implements the `_getitem()` method to retrieve individual data points from the dataset, using the specified `key1` and `key2` for extraction. The returned dictionary contains two text values: `text` (from `self.key1`) and `text_pair` (from `self.key2`).
* **Notable Dependencies**:
	+ Extends `Dataset` class from an unknown library/framework (likely PyTorch, due to the mention of `__len__` methods)
	+ Assumes access to a base dataset object (`dataset`) with compatible indexing and structure.


key: chroma:document
string_value:
CODE:

class TableQuestionAnsweringArgumentHandler(ArgumentHandler):
    """
    Handles arguments for the TableQuestionAnsweringPipeline
    """

    def __call__(self, table=None, query=None, **kwargs):
        # Returns tqa_pipeline_inputs of shape:
        # [
        #   {"table": pd.DataFrame, "query": List[str]},
        #   ...,
        #   {"table": pd.DataFrame, "query" : List[str]}
        # ]
        requires_backends(self, "pandas")
        import pandas as pd

        if table is None:
            raise ValueError("Keyword argument `table` cannot be None.")
        elif query is None:
            if isinstance(table, dict) and table.get("query") is not None and table.get("table") is not None:
                tqa_pipeline_inputs = [table]
            elif isinstance(table, list) and len(table) > 0:
                if not all(isinstance(d, dict) for d in table):
                    raise ValueError(
                        f"Keyword argument `table` should be a list of dict, but is {(type(d) for d in table)}"
                    )

                if table[0].get("query") is not None and table[0].get("table") is not None:
                    tqa_pipeline_inputs = table
                else:
                    raise ValueError(
                        "If keyword argument `table` is a list of dictionaries, each dictionary should have a `table`"
                        f" and `query` key, but only dictionary has keys {table[0].keys()} `table` and `query` keys."
                    )
            elif Dataset is not None and isinstance(table, Dataset) or isinstance(table, types.GeneratorType):
                return table
            else:
                raise ValueError(
                    "Invalid input. Keyword argument `table` should be either of type `dict` or `list`, but "
                    f"is {type(table)})"
                )
        else:
            tqa_pipeline_inputs = [{"table": table, "query": query}]

        for tqa_pipeline_input in tqa_pipeline_inputs:
            if not isinstance(tqa_pipeline_input["table"], pd.DataFrame):
                if tqa_pipeline_input["table"] is None:
                    raise ValueError("Table cannot be None.")

                tqa_pipeline_input["table"] = pd.DataFrame(tqa_pipeline_input["table"])



SUMMARY:
**Class Summary: `TableQuestionAnsweringArgumentHandler`**

*   **Purpose:** Handles arguments for the TableQuestionAnsweringPipeline, transforming tables and queries into a format compatible with TQA pipeline inputs.

### Main Purpose and Functionality

The class takes table and query parameters, ensuring they are valid and formatted correctly. It supports cases where `table` is None or an existing DataFrame, a list of dictionaries (with `table` and `query` keys), or a Dataset object. The resulting transformed tables are stored in the `tqa_pipeline_inputs` variable.

### Key Implementation Details

*   **Data Validation:** Validates input types and formats for `table`, `query`, and `Dataset` objects.
*   **Data Type Conversion:** Converts tables to DataFrames if they are not already.

### Notable Dependencies or Requirements

*   **Pandas (`pandas`):** Required library for data manipulation and processing. All instances of this library are prefixed with `pd.`.

### Classes/Interfaces Implemented

*   Uses the `ArgumentHandler` class interface.
*   Inherently uses the `requires_backends` method (not explicitly implemented here, assuming it's already defined elsewhere in the codebase).

**Key Method Signature:**

```python
def __call__(self, table=None, query=None, **kwargs):
    ...
```

This summary provides an overview of the class's purpose and implementation details while minimizing unnecessary information. The focus is on key concepts, technical terms, method signatures, and notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

def sigmoid(_outputs):


SUMMARY:
**Concise Summary:**

The `_sigmoid` function calculates the Sigmoid activation function:

* Takes an array of `_outputs` as input
* Applies the formula `1 / (1 + exp(-x))`
* Returns the sigmoid of each element in the input array.

### Implementation Details:
* Used for binary classification problems
* Utilizes exponentiation (`exp`) and division operations

### Dependencies:
* `np`: NumPy library is used for the `exp` function.


key: chroma:document
string_value:
CODE:

def softmax(_outputs):
    maxes = np.max(_outputs, axis=-1, keepdims=True)
    shifted_exp = np.exp(_outputs - maxes)


SUMMARY:
**Concise Summary**

### Function Overview

`softmax` calculates the softmax normalization of a set of output values.

### Primary Purpose

The primary purpose is to normalize the input `_outputs` values using the softmax function, which produces a probability distribution over each class or category.

### Key Implementation Details

* The `maxes` variable stores the maximum values along the last axis (-1) of the input `_outputs`.
* The exponential shift (`shifted_exp`) is calculated by subtracting `maxes` from each element in `_outputs`.
* The softmax output is computed by dividing `shifted_exp` by its sum over the last axis (axis=-1).

### Notable Dependencies

* `numpy` (np) for numerical operations.
* `np.max`, `np.exp`, and `np.sum` are used extensively.

### Notes

* This implementation uses NumPy's vectorized operations to compute the softmax output efficiently.
* The `_outputs` variable is expected to be a NumPy array with at least one axis.


key: chroma:document
string_value:
CODE:

class ClassificationFunction(ExplicitEnum):
    SIGMOID = "sigmoid"
    SOFTMAX = "softmax"


SUMMARY:
**Summary:**

The `ClassificationFunction` class defines an explicit enumeration for classification functions, providing three variants:

* `Sigmoid`: Maps to the sigmoid activation function.
* `Softmax`: Maps to the softmax activation function, often used in multi-class classification problems.
* `None`: Represents no transformation or no classification method.

**Key Details:**

* No specific implementation provided; only enumerates possible classification functions.

**Notable Dependencies/Requirements:**

* None explicitly listed, but may be used in conjunction with numerical libraries for statistical calculations (not applicable to this code snippet).


key: chroma:document
string_value:
CODE:

def decode_spans(
    start: np.ndarray, end: np.ndarray, topk: int, max_answer_len: int, undesired_tokens: np.ndarray
) -> Tuple:
    """
    Take the output of any `ModelForQuestionAnswering` and will generate probabilities for each span to be the actual
    answer.

    In addition, it filters out some unwanted/impossible cases like answer len being greater than max_answer_len or
    answer end position being before the starting position. The method supports output the k-best answer through the
    topk argument.

    Args:
        start (`np.ndarray`): Individual start probabilities for each token.
        end (`np.ndarray`): Individual end probabilities for each token.
        topk (`int`): Indicates how many possible answer span(s) to extract from the model output.
        max_answer_len (`int`): Maximum size of the answer to extract from the model's output.
        undesired_tokens (`np.ndarray`): Mask determining tokens that can be part of the answer
    """
    # Ensure we have batch axis
    if start.ndim == 1:
        start = start[None]

    if end.ndim == 1:
        end = end[None]

    # Compute the score of each tuple(start, end) to be the real answer
    outer = np.matmul(np.expand_dims(start, -1), np.expand_dims(end, 1))

    # Remove candidate with end < start and end - start > max_answer_len
    candidates = np.tril(np.triu(outer), max_answer_len - 1)

    #  Inspired by Chen & al. (https://github.com/facebookresearch/DrQA)
    scores_flat = candidates.flatten()
    if topk == 1:
        idx_sort = [np.argmax(scores_flat)]
    elif len(scores_flat) < topk:
        idx_sort = np.argsort(-scores_flat)
    else:
        idx = np.argpartition(-scores_flat, topk)[0:topk]
        idx_sort = idx[np.argsort(-scores_flat[idx])]

    starts, ends = np.unravel_index(idx_sort, candidates.shape)[1:]
    desired_spans = np.isin(starts, undesired_tokens.nonzero()) & np.isin(ends, undesired_tokens.nonzero())
    starts = starts[desired_spans]
    ends = ends[desired_spans]
    scores = candidates[0, starts, ends]



SUMMARY:
**`decode_spans` Function Summary**

**Purpose and Functionality:**
This function decodes the output of a `ModelForQuestionAnswering` by generating probabilities for each span to be the actual answer. It filters out unwanted cases such as incorrect answer length or token positions.

**Key Implementation Details:**

* Computation of score between `start` and `end` tokens using matrix multiplication.
* Filtering of candidates based on invalid end positions (before start) or exceeding maximum answer length.
* Sorting of top-k candidate spans by their scores using NumPy's `argpartition` and `argsort` functions.

**Notable Dependencies and Requirements:**

* `np.ndarray`: Input arrays for start, end, and undesired tokens.
* `ModelForQuestionAnswering`: Output data type not explicitly specified in the function.
* No external dependencies listed.

**Return Value:**
A tuple containing the starts, ends, and scores of the top-k candidate spans.


key: chroma:document
string_value:
CODE:

def select_starts_ends(
    start,
    end,
    p_mask,
    attention_mask,
    min_null_score=1000000,
    top_k=1,
    handle_impossible_answer=False,
    max_answer_len=15,
):
    """
    Takes the raw output of any `ModelForQuestionAnswering` and first normalizes its outputs and then uses
    `decode_spans()` to generate probabilities for each span to be the actual answer.

    Args:
        start (`np.ndarray`): Individual start logits for each token.
        end (`np.ndarray`): Individual end logits for each token.
        p_mask (`np.ndarray`): A mask with 1 for values that cannot be in the answer
        attention_mask (`np.ndarray`): The attention mask generated by the tokenizer
        min_null_score(`float`): The minimum null (empty) answer score seen so far.
        topk (`int`): Indicates how many possible answer span(s) to extract from the model output.
        handle_impossible_answer(`bool`): Whether to allow null (empty) answers
        max_answer_len (`int`): Maximum size of the answer to extract from the model's output.
    """
    # Ensure padded tokens & question tokens cannot belong to the set of candidate answers.
    undesired_tokens = np.abs(np.array(p_mask) - 1)

    if attention_mask is not None:
        undesired_tokens = undesired_tokens & attention_mask

    # Generate mask
    undesired_tokens_mask = undesired_tokens == 0.0

    # Make sure non-context indexes in the tensor cannot contribute to the softmax
    start = np.where(undesired_tokens_mask, -10000.0, start)
    end = np.where(undesired_tokens_mask, -10000.0, end)

    # Normalize logits and spans to retrieve the answer
    start = np.exp(start - start.max(axis=-1, keepdims=True))
    start = start / start.sum()

    end = np.exp(end - end.max(axis=-1, keepdims=True))
    end = end / end.sum()

    if handle_impossible_answer:
        min_null_score = min(min_null_score, (start[0, 0] * end[0, 0]).item())

    # Mask CLS
    start[0, 0] = end[0, 0] = 0.0

    starts, ends, scores = decode_spans(start, end, top_k, max_answer_len, undesired_tokens)


SUMMARY:
**Function Summary: `select_starts_ends`**

**Purpose and Functionality**: This function takes the raw output of a `ModelForQuestionAnswering` and normalizes it to generate probabilities for each span to be the actual answer.

**Key Implementation Details**:

* Normalizes start and end logits using softmax, excluding undesired tokens.
* Applies additional smoothing by setting start and end masks to 0.0 at CLS token position.
* Uses `decode_spans()` function to extract top-k possible answer spans.

**Notable Dependencies and Requirements**:

* `np.ndarray` inputs for raw output
* External dependency on `ModelForQuestionAnswering`
* `decode_spans()` function, not defined in this code chunk.


key: chroma:document
string_value:
CODE:

class QuestionAnsweringArgumentHandler(ArgumentHandler):
    """
    QuestionAnsweringPipeline requires the user to provide multiple arguments (i.e. question & context) to be mapped to
    internal [`SquadExample`].

    QuestionAnsweringArgumentHandler manages all the possible to create a [`SquadExample`] from the command-line
    supplied arguments.
    """

    def normalize(self, item):
        if isinstance(item, SquadExample):
            return item
        elif isinstance(item, dict):
            for k in ["question", "context"]:
                if k not in item:
                    raise KeyError("You need to provide a dictionary with keys {question:..., context:...}")
                elif item[k] is None:
                    raise ValueError(f"`{k}` cannot be None")
                elif isinstance(item[k], str) and len(item[k]) == 0:
                    raise ValueError(f"`{k}` cannot be empty")

            return QuestionAnsweringPipeline.create_sample(**item)
        raise ValueError(f"{item} argument needs to be of type (SquadExample, dict)")

    def __call__(self, *args, **kwargs):
        # Detect where the actual inputs are
        if args is not None and len(args) > 0:
            if len(args) == 1:
                inputs = args[0]
            elif len(args) == 2 and {type(el) for el in args} == {str}:
                inputs = [{"question": args[0], "context": args[1]}]
            else:
                inputs = list(args)
        # Generic compatibility with sklearn and Keras
        # Batched data
        elif "X" in kwargs:
            warnings.warn(
                "Passing the `X` argument to the pipeline is deprecated and will be removed in v5. Inputs should be passed using the `question` and `context` keyword arguments instead.",
                FutureWarning,
            )
            inputs = kwargs["X"]
        elif "data" in kwargs:
            warnings.warn(
                "Passing the `data` argument to the pipeline is deprecated and will be removed in v5. Inputs should be passed using the `question` and `context` keyword arguments instead.",
                FutureWarning,
            )
            inputs = kwargs["data"]
        elif "question" in kwargs and "context" in kwargs:
            if isinstance(kwargs["question"], list) and isinstance(kwargs["context"], str):
                inputs = [{"question": Q, "context": kwargs["context"]} for Q in kwargs["question"]]
            elif isinstance(kwargs["question"], list) and isinstance(kwargs["context"], list):
                if len(kwargs["question"]) != len(kwargs["context"]):
                    raise ValueError("Questions and contexts don't have the same lengths")

                inputs = [{"question": Q, "context": C} for Q, C in zip(kwargs["question"], kwargs["context"])]
            elif isinstance(kwargs["question"], str) and isinstance(kwargs["context"], str):
                inputs = [{"question": kwargs["question"], "context": kwargs["context"]}]
            else:
                raise ValueError("Arguments can't be understood")
        else:
            raise ValueError(f"Unknown arguments {kwargs}")

        # When user is sending a generator we need to trust it's a valid example
        generator_types = (types.GeneratorType, Dataset) if Dataset is not None else (types.GeneratorType,)
        if isinstance(inputs, generator_types):
            return inputs

        # Normalize inputs
        if isinstance(inputs, dict):
            inputs = [inputs]
        elif isinstance(inputs, Iterable):
            # Copy to avoid overriding arguments
            inputs = list(inputs)
        else:
            raise ValueError(f"Invalid arguments {kwargs}")

        for i, item in enumerate(inputs):
            inputs[i] = self.normalize(item)



SUMMARY:
**QuestionAnsweringArgumentHandler Class Summary**

### Overview
The `QuestionAnsweringArgumentHandler` class is responsible for managing the input arguments and creating a `SquadExample` from them. It inherits from the `ArgumentHandler` class.

### Primary Purpose
To normalize the input arguments (question, context, or data) and return a list of normalized `SquadExample` instances.

### Key Implementation Details

*   **normalize method**: This method takes an argument and checks if it's a `SquadExample`, dictionary, question, context, generator, or iterable. It raises errors for invalid arguments.
*   **__call__ method**: This method detects the input sources (args or kwargs) and passes them to the `normalize` method for processing.

### Notable Dependencies or Requirements

*   The class uses the `types.GeneratorType` and `Dataset` classes from Python 3's `typing` module. It assumes that a generator is an instance of these types.
*   The class also uses external dependencies such as `SquadExample` (not defined in this snippet), `ArgumentHandler`, `FutureWarning`, and `sklearn` or Keras, which are not explicitly imported.

### Interfaces and Inherited Methods

The class inherits from the `ArgumentHandler` base class and does not implement any new interfaces. However, it overrides the built-in object behavior to use its `__call__` method as a callable instance.


key: chroma:document
string_value:
CODE:

def ffmpeg_read(bpayload: bytes, sampling_rate: int) -> np.array:
    """
    Helper function to read an audio file through ffmpeg.
    """
    ar = f"{sampling_rate}"
    ac = "1"
    format_for_conversion = "f32le"
    ffmpeg_command = [
        "ffmpeg",
        "-i",
        "pipe:0",
        "-ac",
        ac,
        "-ar",
        ar,
        "-f",
        format_for_conversion,
        "-hide_banner",
        "-loglevel",
        "quiet",
        "pipe:1",
    ]

    try:
        with subprocess.Popen(ffmpeg_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE) as ffmpeg_process:
            output_stream = ffmpeg_process.communicate(bpayload)
    except FileNotFoundError as error:
        raise ValueError("ffmpeg was not found but is required to load audio files from filename") from error
    out_bytes = output_stream[0]
    audio = np.frombuffer(out_bytes, np.float32)
    if audio.shape[0] == 0:
        raise ValueError(
            "Soundfile is either not in the correct format or is malformed. Ensure that the soundfile has "
            "a valid audio file extension (e.g. wav, flac or mp3) and is not corrupted. If reading from a remote "
            "URL, ensure that the URL is the full address to **download** the audio file."
        )


SUMMARY:
**FFmpeg Audio Reader Function Summary**

**Purpose**: Reads an audio file through FFmpeg and returns the audio data as a NumPy array.

**Implementation Details**:

* Uses `ffmpeg` command-line tool to read raw audio data from input bytes (`bpayload`) with specified sampling rate (`sampling_rate`)
* Converts resulting stream from FFmpeg into a NumPy float32 array using `np.frombuffer`

**Key Algorithm/Design Choices**:

* Utilizes FFmpeg's `-ac`, `-ar`, and `-f` options to configure audio conversion
* Leverages the `pipe:0` and `pipe:1` syntax for input/output redirection

**Notable Dependencies/Requirements**:

* FFmpeg executable is required to be installed on the system, which raises a `ValueError` if not found (`subprocess.Popen` raise exception)

**Additional Notes**: Input audio data must conform to standard formats (e.g. wav, flac) and contain usable audio data to avoid raising errors.
```python
def ffmpeg_read(bpayload: bytes, sampling_rate: int) -> np.array:
    # FFmpeg audio reader function implementation details...
    
    return audio
```
**External Dependencies:**

* `ffmpeg` command-line tool
* NumPy for audio data processing


key: chroma:document
string_value:
CODE:

def ffmpeg_microphone(
    sampling_rate: int,
    chunk_length_s: float,
    format_for_conversion: str = "f32le",
    ffmpeg_input_device: Optional[str] = None,
    ffmpeg_additional_args: Optional[list[str]] = None,
):
    """
    Helper function to read audio from a microphone using ffmpeg. The default input device will be used unless another
    input device is specified using the `ffmpeg_input_device` argument. Uses 'alsa' on Linux, 'avfoundation' on MacOS and
    'dshow' on Windows.

    Arguments:
        sampling_rate (`int`):
            The sampling_rate to use when reading the data from the microphone. Try using the model's sampling_rate to
            avoid resampling later.
        chunk_length_s (`float` or `int`):
            The length of the maximum chunk of audio to be sent returned.
        format_for_conversion (`str`, defaults to `f32le`):
            The name of the format of the audio samples to be returned by ffmpeg. The standard is `f32le`, `s16le`
            could also be used.
        ffmpeg_input_device (`str`, *optional*):
            The identifier of the input device to be used by ffmpeg (i.e. ffmpeg's '-i' argument). If unset,
            the default input device will be used. See `https://www.ffmpeg.org/ffmpeg-devices.html#Input-Devices`
            for how to specify and list input devices.
        ffmpeg_additional_args (`list[str]`, *optional*):
            Additional arguments to pass to ffmpeg, can include arguments like -nostdin for running as a background
            process. For example, to pass -nostdin to the ffmpeg process, pass in ["-nostdin"]. If passing in flags
            with multiple arguments, use the following convention (eg ["flag", "arg1", "arg2]).

    Returns:
        A generator yielding audio chunks of `chunk_length_s` seconds as `bytes` objects of length
        `int(round(sampling_rate * chunk_length_s)) * size_of_sample`.
    """
    ar = f"{sampling_rate}"
    ac = "1"
    if format_for_conversion == "s16le":
        size_of_sample = 2
    elif format_for_conversion == "f32le":
        size_of_sample = 4
    else:
        raise ValueError(f"Unhandled format `{format_for_conversion}`. Please use `s16le` or `f32le`")

    system = platform.system()

    if system == "Linux":
        format_ = "alsa"
        input_ = ffmpeg_input_device or "default"
    elif system == "Darwin":
        format_ = "avfoundation"
        input_ = ffmpeg_input_device or ":default"
    elif system == "Windows":
        format_ = "dshow"
        input_ = ffmpeg_input_device or _get_microphone_name()

    ffmpeg_additional_args = [] if ffmpeg_additional_args is None else ffmpeg_additional_args

    ffmpeg_command = [
        "ffmpeg",
        "-f",
        format_,
        "-i",
        input_,
        "-ac",
        ac,
        "-ar",
        ar,
        "-f",
        format_for_conversion,
        "-fflags",
        "nobuffer",
        "-hide_banner",
        "-loglevel",
        "quiet",
        "pipe:1",
    ]

    ffmpeg_command.extend(ffmpeg_additional_args)

    chunk_len = int(round(sampling_rate * chunk_length_s)) * size_of_sample
    iterator = _ffmpeg_stream(ffmpeg_command, chunk_len)
    for item in iterator:


SUMMARY:
**Concise Summary:**
```python
**Function:** `ffmpeg_microphone`

The `ffmpeg_microphone` function reads audio from a microphone using the FFmpeg library. It takes several parameters, including sampling rate, chunk length, format for conversion, and additional FFmpeg arguments.

**Main Purpose and Functionality:**

* Reads audio from a microphone using FF McMahon
* Supports multiple platforms (Linux, MacOS, Windows) with device-specific configurations
* Produces generator yielding audio chunks of specified length

**Key Implementation Details:**

* Uses FFmpeg to convert audio sample format (e.g., 16-bit or 32-bit floating-point)
* Sets input device and arguments using platform-specific conventions
* Returns a generator yieldings audio chunks as `bytes` objects

**Notable Dependencies and Requirements:**

* FFmpeg library
* Platform-specific configuration for input devices and command-line options
```

Note that I've highlighted the main purpose, key implementation details, and notable dependencies/requirements in a concise format. Let me know if you'd like me to elaborate on any aspect!


key: chroma:document
string_value:
CODE:

def ffmpeg_microphone_live(
    sampling_rate: int,
    chunk_length_s: float,
    stream_chunk_s: Optional[int] = None,
    stride_length_s: Optional[Union[Tuple[float, float], float]] = None,
    format_for_conversion: str = "f32le",
    ffmpeg_input_device: Optional[str] = None,
    ffmpeg_additional_args: Optional[list[str]] = None,
):
    """
    Helper function to read audio from a microphone using ffmpeg. This will output `partial` overlapping chunks starting
    from `stream_chunk_s` (if it is defined) until `chunk_length_s` is reached. It will make use of striding to avoid
    errors on the "sides" of the various chunks. The default input device will be used unless another input device is
    specified using the `ffmpeg_input_device` argument. Uses 'alsa' on Linux, 'avfoundation' on MacOS and 'dshow' on Windows.

    Arguments:
        sampling_rate (`int`):
            The sampling_rate to use when reading the data from the microphone. Try using the model's sampling_rate to
            avoid resampling later.
        chunk_length_s (`float` or `int`):
            The length of the maximum chunk of audio to be sent returned. This includes the eventual striding.
        stream_chunk_s (`float` or `int`):
            The length of the minimal temporary audio to be returned.
        stride_length_s (`float` or `int` or `(float, float)`, *optional*):
            The length of the striding to be used. Stride is used to provide context to a model on the (left, right) of
            an audio sample but without using that part to actually make the prediction. Setting this does not change
            the length of the chunk.
        format_for_conversion (`str`, *optional*, defaults to `f32le`):
            The name of the format of the audio samples to be returned by ffmpeg. The standard is `f32le`, `s16le`
            could also be used.
        ffmpeg_input_device (`str`, *optional*):
            The identifier of the input device to be used by ffmpeg (i.e. ffmpeg's '-i' argument). If unset,
            the default input device will be used. See `https://www.ffmpeg.org/ffmpeg-devices.html#Input-Devices`
            for how to specify and list input devices.
        ffmpeg_additional_args (`list[str]`, *optional*):
            Additional arguments to pass to ffmpeg, can include arguments like -nostdin for running as a background
            process. For example, to pass -nostdin to the ffmpeg process, pass in ["-nostdin"]. If passing in flags
            with multiple arguments, use the following convention (eg ["flag", "arg1", "arg2]).

    Return:
        A generator yielding dictionaries of the following form

        `{"sampling_rate": int, "raw": np.array(), "partial" bool}` With optionally a `"stride" (int, int)` key if
        `stride_length_s` is defined.

        `stride` and `raw` are all expressed in `samples`, and `partial` is a boolean saying if the current yield item
        is a whole chunk, or a partial temporary result to be later replaced by another larger chunk.
    """
    if stream_chunk_s is not None:
        chunk_s = stream_chunk_s
    else:
        chunk_s = chunk_length_s

    microphone = ffmpeg_microphone(
        sampling_rate,
        chunk_s,
        format_for_conversion=format_for_conversion,
        ffmpeg_input_device=ffmpeg_input_device,
        ffmpeg_additional_args=[] if ffmpeg_additional_args is None else ffmpeg_additional_args,
    )

    if format_for_conversion == "s16le":
        dtype = np.int16
        size_of_sample = 2
    elif format_for_conversion == "f32le":
        dtype = np.float32
        size_of_sample = 4
    else:
        raise ValueError(f"Unhandled format `{format_for_conversion}`. Please use `s16le` or `f32le`")

    if stride_length_s is None:
        stride_length_s = chunk_length_s / 6
    chunk_len = int(round(sampling_rate * chunk_length_s)) * size_of_sample
    if isinstance(stride_length_s, (int, float)):
        stride_length_s = [stride_length_s, stride_length_s]

    stride_left = int(round(sampling_rate * stride_length_s[0])) * size_of_sample
    stride_right = int(round(sampling_rate * stride_length_s[1])) * size_of_sample
    audio_time = datetime.datetime.now()
    delta = datetime.timedelta(seconds=chunk_s)
    for item in chunk_bytes_iter(microphone, chunk_len, stride=(stride_left, stride_right), stream=True):
        # Put everything back in numpy scale
        item["raw"] = np.frombuffer(item["raw"], dtype=dtype)
        item["stride"] = (
            item["stride"][0] // size_of_sample,
            item["stride"][1] // size_of_sample,
        )
        item["sampling_rate"] = sampling_rate
        audio_time += delta
        if datetime.datetime.now() > audio_time + 10 * delta:
            # We're late !! SKIP
            continue


SUMMARY:
**Summary: FFMPEG Microphone Live Function**

### Purpose and Functionality

The `ffmpeg_microphone_live` function reads audio from a microphone using FFmpeg, producing overlapping chunks of audio starting from a specified length (`stream_chunk_s`) until reaching the maximum chunk length (`chunk_length_s`). It uses stride to provide context to the model without utilizing the side samples.

### Key Implementation Details

*   Uses `ffmpeg_microphone` helper function to initialize the audio stream.
*   Determines the format for conversion based on the input format (`f32le` or `s16le`).
*   Calculates the stride length using the sampling rate and chunk length.
*   Iterates over the data chunks, yielding dictionaries containing the raw audio data, stride information, and other metadata.

### Dependencies and Requirements

*   `ffmpeg` library (not explicitly imported in this code snippet)
*   NumPy (`np`) for handling numerical data
*   FFmpeg device libraries for Linux («alsa»), MacOS («avfoundation»), and Windows («dshow»).

### Method Signatures and Interfaces

`ffmpeg_microphone_live` takes the following parameters:
    -   `sampling_rate`: int
    -   `chunk_length_s`: float or int
    -   `stream_chunk_s`: optional float or int
    -   `stride_length_s`: optional tuple of floats or a single float, defaults to 6.0x (or 60 seconds)
    -   `format_for_conversion`: str, default `'f32le'`
    -   `ffmpeg_input_device`: optional str
    -   `ffmpeg_additional_args`: optional list[str]

The function returns a generator yielding dictionaries containing:
    -   `"sampling_rate"`: int
    -   `"raw"`: NumPy array
    -   `"partial"`: boolean indicating whether the result is whole or partial


key: chroma:document
string_value:
CODE:

def chunk_bytes_iter(iterator, chunk_len: int, stride: Tuple[int, int], stream: bool = False):
    """
    Reads raw bytes from an iterator and does chunks of length `chunk_len`. Optionally adds `stride` to each chunks to
    get overlaps. `stream` is used to return partial results even if a full `chunk_len` is not yet available.
    """
    acc = b""
    stride_left, stride_right = stride
    if stride_left + stride_right >= chunk_len:
        raise ValueError(
            f"Stride needs to be strictly smaller than chunk_len: ({stride_left}, {stride_right}) vs {chunk_len}"
        )
    _stride_left = 0
    for raw in iterator:
        acc += raw
        if stream and len(acc) < chunk_len:
            stride = (_stride_left, 0)
            yield {"raw": acc[:chunk_len], "stride": stride, "partial": True}
        else:
            while len(acc) >= chunk_len:
                # We are flushing the accumulator
                stride = (_stride_left, stride_right)
                item = {"raw": acc[:chunk_len], "stride": stride}
                if stream:
                    item["partial"] = False
                yield item
                _stride_left = stride_left
                acc = acc[chunk_len - stride_left - stride_right :]
    # Last chunk
    if len(acc) > stride_left:
        item = {"raw": acc, "stride": (_stride_left, 0)}
        if stream:
            item["partial"] = False


SUMMARY:
**Function Summary: `chunk_bytes_iter`**

**Purpose:** Streams bytes from an iterator in chunks of length `chunk_len`, optionally with stride for overlap.

```python
def chunk_bytes_iter(iterator, chunk_len: int, stride: Tuple[int, int], stream: bool = False):
```

**Key Implementation Details:**

* **Chunking**: Accumulates bytes and yields full chunks (`{raw: ..., len >= chunk_len}`). 
* **Partial Results**: Optionally returns partial results even if a full `chunk_len` is not yet available.

```python
acc += raw
if stream and len(acc) < chunk_len:
    yield {"raw": acc[:chunk_len], "stride": (_stride_left, 0), "partial": True}
else:
yield {"raw": acc[:chunk_len], "stride": (_stride_left, stride_right)}
```

**Notable Dependencies and Requirements:**

* `iterator`: iterator of raw bytes
* External dependencies: None

**Interfaces and Inheritance:**
None


key: chroma:document
string_value:
CODE:

def _ffmpeg_stream(ffmpeg_command, buflen: int):
    """
    Internal function to create the generator of data through ffmpeg
    """
    bufsize = 2**24  # 16Mo
    try:
        with subprocess.Popen(ffmpeg_command, stdout=subprocess.PIPE, bufsize=bufsize) as ffmpeg_process:
            while True:
                raw = ffmpeg_process.stdout.read(buflen)
                if raw == b"":
                    break
                yield raw
    except FileNotFoundError as error:


SUMMARY:
**Summary**

The `_ffmpeg_stream` function creates a generator that produces audio data through the FFmpeg process. It takes an `ffmpeg_command` and a buffer size (`buflen`) as input.

**Key Implementation Details**

- **FFmpeg Command**: Executes a custom FFmpeg command using `subprocess.Popen`.
- **Buffer Size**: Uses a static buffer size of 16MB (`bufsize = 2**24`).
- **Data Streaming**: Yields raw audio data from the FFmpeg process in chunks of `buflen`.

**Notable Dependencies and Requirements**

- **FFmpeg**: The function relies on the presence of FFmpeg, which will throw a `ValueError` if not found.
- **Subprocess**: Uses the `Popen` class to execute the FFmpeg command.
- **Buffering**: Does not inherit or implement any interfaces; uses fixed-size buffering (default buffer size).


key: chroma:document
string_value:
CODE:

def _get_microphone_name():
    """
    Retrieve the microphone name in Windows .
    """
    command = ["ffmpeg", "-list_devices", "true", "-f", "dshow", "-i", ""]

    try:
        ffmpeg_devices = subprocess.run(command, text=True, stderr=subprocess.PIPE, encoding="utf-8")
        microphone_lines = [line for line in ffmpeg_devices.stderr.splitlines() if "(audio)" in line]

        if microphone_lines:
            microphone_name = microphone_lines[0].split('"')[1]
            print(f"Using microphone: {microphone_name}")
            return f"audio={microphone_name}"
    except FileNotFoundError:
        print("ffmpeg was not found. Please install it or make sure it is in your system PATH.")



SUMMARY:
**Summary**

### Function Purpose and Main Functionality

This function, `_get_microphone_name()`, retrieves the name of the selected microphone using FFmpeg's `dshow` device filter. It prints the used microphone name and returns a string indicating audio input with the microphone name.

### Key Implementation Details

* The command `ffmpeg `-list_devices true -f dshow -i ""` lists all available capture devices.
* A list comprehension iterates over the error output to extract lines containing "(audio)".
* The first valid line is parsed to obtain the microphone's name, which is then printed and returned.

### Dependencies and Requirements

* `ffmpeg` executable is required for execution; if not found, an error message is printed suggesting installation or system PATH addition.


key: docstring
string_value:
# normalize_bbox() and apply_tesseract() are derived from apply_tesseract in models/layoutlmv3/feature_extraction_layoutlmv3.py.
# However, because the pipeline may evolve from what layoutlmv3 currently does, it's copied (vs. imported) to avoid creating an
# unnecessary dependency.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/src/transformers/pipelines/document_question_answering.py


key: chroma:document
string_value:
CODE:
# unnecessary dependency.
def normalize_box(box, width, height):
    return [
        int(1000 * (box[0] / width)),
        int(1000 * (box[1] / height)),
        int(1000 * (box[2] / width)),
        int(1000 * (box[3] / height)),


SUMMARY:
### Function Summary: `normalize_box`

#### Purpose and Functionality

Normalizes a bounding box by scaling coordinates to the range -1 to 1.

`normalize_box(box, width, height)` converts input box coordinates from the original image dimensions (`width`, `height`) to a standard range (-1, 1).

#### Key Implementation Details

- Scales x and y coordinates by a factor of 1000 relative to their respective input dimensions.
- Returns a list containing normalized x, y, and two (x and y) scale factors.

#### Notable Dependencies or Requirements

- `apply_tesseract`: The normalize_box() function is derived from this model, suggesting potential compatibility issues with changes to the pipeline if this function is not maintained locally.
- No external dependencies listed.


key: chroma:document
string_value:
CODE:

def apply_tesseract(image: "Image.Image", lang: Optional[str], tesseract_config: Optional[str]):
    """Applies Tesseract OCR on a document image, and returns recognized words + normalized bounding boxes."""
    # apply OCR
    data = pytesseract.image_to_data(image, lang=lang, output_type="dict", config=tesseract_config)
    words, left, top, width, height = data["text"], data["left"], data["top"], data["width"], data["height"]

    # filter empty words and corresponding coordinates
    irrelevant_indices = [idx for idx, word in enumerate(words) if not word.strip()]
    words = [word for idx, word in enumerate(words) if idx not in irrelevant_indices]
    left = [coord for idx, coord in enumerate(left) if idx not in irrelevant_indices]
    top = [coord for idx, coord in enumerate(top) if idx not in irrelevant_indices]
    width = [coord for idx, coord in enumerate(width) if idx not in irrelevant_indices]
    height = [coord for idx, coord in enumerate(height) if idx not in irrelevant_indices]

    # turn coordinates into (left, top, left+width, top+height) format
    actual_boxes = []
    for x, y, w, h in zip(left, top, width, height):
        actual_box = [x, y, x + w, y + h]
        actual_boxes.append(actual_box)

    image_width, image_height = image.size

    # finally, normalize the bounding boxes
    normalized_boxes = []
    for box in actual_boxes:
        normalized_boxes.append(normalize_box(box, image_width, image_height))

    if len(words) != len(normalized_boxes):
        raise ValueError("Not as many words as there are bounding boxes")



SUMMARY:
**Function Summary: Apply Tesseract OCR on Document Image**

* **Purpose:** Applies Tesseract Optical Character Recognition (OCR) on a document image, extracting recognized words and normalizing their corresponding bounding boxes.
* **Main Functionality:**
	+ Uses `pytesseract.image_to_data` to apply OCR on the provided image with specified language and configuration settings.
	+ Filters out empty words and their coordinates from the result.
	+ Normalizes the remaining coordinates into the (left, top, left+width, top+height) format for bounding boxes.

**Key Implementation Details:**

* `pytesseract.image_to_data` function call with `output_type="dict"` parameter.
* Filtering out empty words and their coordinates using list comprehensions.
* Normalization of coordinates using the `normalize_box` function (not shown in this snippet).

**Dependencies and Requirements:**

* `pytesseract` library for Tesseract OCR functionality.
* `Image` class from PIL library for image processing.
* `Optional` type hint from typing library for handling language and configuration parameters.

**Method Signatures and Interfaces:**

* `apply_tesseract` function takes three parameters:
	+ `image`: an instance of the `Image` class.
	+ `lang`: a string representing the language for OCR.
	+ `tesseract_config`: a string representing the Tesseract configuration settings.


key: chroma:document
string_value:
CODE:

class ModelType(ExplicitEnum):
    LayoutLM = "layoutlm"
    LayoutLMv2andv3 = "layoutlmv2andv3"


SUMMARY:
Here is a concise summary of the code:

**ModelType Class Summary**

The `ModelType` class defines an enumeration (ExplicitEnum) with three values: `LayoutLM`, `LayoutLMv2andv3`, and `VisionEncoderDecoder`. These values represent different model configurations, specifically tailored for Natural Language Processing (NLP) tasks.


key: chroma:document
string_value:
CODE:
# Copied from transformers.pipelines.text_classification.sigmoid
def sigmoid(_outputs):


SUMMARY:
**Summary**

The `sigmoid` function applies the sigmoid activation function to a set of `_outputs`. It re-imagines NumPy's exponential function with an addition term, producing a probability value between 0 and 1.

*   **Key Implementation Details:**
    *   Sigmoid activation function.
    *   Implements sigmoid function using numerical stability optimizations.
*   **Method Signatures and Interfaces:** `_sigmoid` (not explicitly defined) is expected to take one argument `_outputs`.
*   **Notable Dependencies or Requirements:**

        *   `np`: NumPy library for vectorized operations.


key: chroma:document
string_value:
CODE:
# Copied from transformers.pipelines.text_classification.softmax
def softmax(_outputs):
    maxes = np.max(_outputs, axis=-1, keepdims=True)
    shifted_exp = np.exp(_outputs - maxes)


SUMMARY:
**Concise Summary**

`softmax(_outputs)` computes the softmax activation function applied to a set of _outputs_.

**Key Implementation Details**

* Computes maximum values along the last axis (axis=-1) to normalize outputs.
* Shifts and exponentiates outputs by subtracting maximum values.
* Normalizes exponents by summing them along the last axis.
* Returns a 2D array with softmax-activated values.

**Notable Dependencies/Requirements**

* NumPy (`np`) is required for numerical computations.


key: chroma:document
string_value:
CODE:
# Copied from transformers.pipelines.text_classification.ClassificationFunction
class ClassificationFunction(ExplicitEnum):
    SIGMOID = "sigmoid"
    SOFTMAX = "softmax"


SUMMARY:
**Summary**

`ClassificationFunction` Enum:
A specialized enumeration for classification tasks in NLP, providing three predefined activation functions for text classification:

- `SIGMOID`: Sigmoid function
- `SOFTMAX`: Softmax function
- `NONE`: No activation applied (default)

**Key Implementation Details**

* Note: The implementation details are not provided here as they belong to the referenced `transformers.pipelines.text_classification.ClassificationFunction` class.

**Notable Dependencies/Requirements**

* `ExplicitEnum`: A base class for creating explicit enumerations.
* Transformers' text classification pipeline: The classification function is part of this library.


key: chroma:document
string_value:
CODE:

class TextToAudioPipeline(Pipeline):
    """
    Text-to-audio generation pipeline using any `AutoModelForTextToWaveform` or `AutoModelForTextToSpectrogram`. This
    pipeline generates an audio file from an input text and optional other conditional inputs.

    Example:

    ```python
    >>> from transformers import pipeline

    >>> pipe = pipeline(model="suno/bark-small")
    >>> output = pipe("Hey it's HuggingFace on the phone!")

    >>> audio = output["audio"]
    >>> sampling_rate = output["sampling_rate"]
    ```

    Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)

    <Tip>

    You can specify parameters passed to the model by using [`TextToAudioPipeline.__call__.forward_params`] or
    [`TextToAudioPipeline.__call__.generate_kwargs`].

    Example:

    ```python
    >>> from transformers import pipeline

    >>> music_generator = pipeline(task="text-to-audio", model="facebook/musicgen-small", framework="pt")

    >>> # diversify the music generation by adding randomness with a high temperature and set a maximum music length
    >>> generate_kwargs = {
    ...     "do_sample": True,
    ...     "temperature": 0.7,
    ...     "max_new_tokens": 35,
    ... }

    >>> outputs = music_generator("Techno music with high melodic riffs", generate_kwargs=generate_kwargs)
    ```

    </Tip>

    This pipeline can currently be loaded from [`pipeline`] using the following task identifiers: `"text-to-speech"` or
    `"text-to-audio"`.

    See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=text-to-speech).
    """

    def __init__(self, *args, vocoder=None, sampling_rate=None, **kwargs):
        super().__init__(*args, **kwargs)

        if self.framework == "tf":
            raise ValueError("The TextToAudioPipeline is only available in PyTorch.")

        self.vocoder = None
        if self.model.__class__ in MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING.values():
            self.vocoder = (
                SpeechT5HifiGan.from_pretrained(DEFAULT_VOCODER_ID).to(self.model.device)
                if vocoder is None
                else vocoder
            )

        self.sampling_rate = sampling_rate
        if self.vocoder is not None:
            self.sampling_rate = self.vocoder.config.sampling_rate

        if self.sampling_rate is None:
            # get sampling_rate from config and generation config

            config = self.model.config
            gen_config = self.model.__dict__.get("generation_config", None)
            if gen_config is not None:
                config.update(gen_config.to_dict())

            for sampling_rate_name in ["sample_rate", "sampling_rate"]:
                sampling_rate = getattr(config, sampling_rate_name, None)
                if sampling_rate is not None:
                    self.sampling_rate = sampling_rate

    def preprocess(self, text, **kwargs):
        if isinstance(text, str):
            text = [text]

        if self.model.config.model_type == "bark":
            # bark Tokenizer is called with BarkProcessor which uses those kwargs
            new_kwargs = {
                "max_length": self.generation_config.semantic_config.get("max_input_semantic_length", 256),
                "add_special_tokens": False,
                "return_attention_mask": True,
                "return_token_type_ids": False,
                "padding": "max_length",
            }

            # priority is given to kwargs
            new_kwargs.update(kwargs)

            kwargs = new_kwargs

        output = self.tokenizer(text, **kwargs, return_tensors="pt")

        return output

    def _forward(self, model_inputs, **kwargs):
        # we expect some kwargs to be additional tensors which need to be on the right device
        kwargs = self._ensure_tensor_on_device(kwargs, device=self.device)
        forward_params = kwargs["forward_params"]
        generate_kwargs = kwargs["generate_kwargs"]

        if self.model.can_generate():
            # we expect some kwargs to be additional tensors which need to be on the right device
            generate_kwargs = self._ensure_tensor_on_device(generate_kwargs, device=self.device)

            # User-defined `generation_config` passed to the pipeline call take precedence
            if "generation_config" not in generate_kwargs:
                generate_kwargs["generation_config"] = self.generation_config

            # generate_kwargs get priority over forward_params
            forward_params.update(generate_kwargs)

            output = self.model.generate(**model_inputs, **forward_params)
        else:
            if len(generate_kwargs):
                raise ValueError(
                    "You're using the `TextToAudioPipeline` with a forward-only model, but `generate_kwargs` is non "
                    "empty. For forward-only TTA models, please use `forward_params` instead of `generate_kwargs`. "
                    f"For reference, the `generate_kwargs` used here are: {generate_kwargs.keys()}"
                )
            output = self.model(**model_inputs, **forward_params)[0]

        if self.vocoder is not None:
            # in that case, the output is a spectrogram that needs to be converted into a waveform
            output = self.vocoder(output)

        return output

    def __call__(self, text_inputs: Union[str, List[str]], **forward_params):
        """
        Generates speech/audio from the inputs. See the [`TextToAudioPipeline`] documentation for more information.

        Args:
            text_inputs (`str` or `List[str]`):
                The text(s) to generate.
            forward_params (`dict`, *optional*):
                Parameters passed to the model generation/forward method. `forward_params` are always passed to the
                underlying model.
            generate_kwargs (`dict`, *optional*):
                The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a
                complete overview of generate, check the [following
                guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). `generate_kwargs` are
                only passed to the underlying model if the latter is a generative model.

        Return:
            A `dict` or a list of `dict`: The dictionaries have two keys:

            - **audio** (`np.ndarray` of shape `(nb_channels, audio_length)`) -- The generated audio waveform.
            - **sampling_rate** (`int`) -- The sampling rate of the generated audio waveform.
        """
        return super().__call__(text_inputs, **forward_params)

    def _sanitize_parameters(
        self,
        preprocess_params=None,
        forward_params=None,
        generate_kwargs=None,
    ):
        if self.assistant_model is not None:
            generate_kwargs["assistant_model"] = self.assistant_model
        if self.assistant_tokenizer is not None:
            generate_kwargs["tokenizer"] = self.tokenizer
            generate_kwargs["assistant_tokenizer"] = self.assistant_tokenizer

        params = {
            "forward_params": forward_params if forward_params else {},
            "generate_kwargs": generate_kwargs if generate_kwargs else {},
        }

        if preprocess_params is None:
            preprocess_params = {}
        postprocess_params = {}

        return preprocess_params, params, postprocess_params

    def postprocess(self, waveform):
        output_dict = {}
        if isinstance(waveform, dict):
            waveform = waveform["waveform"]
        elif isinstance(waveform, tuple):
            waveform = waveform[0]
        output_dict["audio"] = waveform.cpu().float().numpy()
        output_dict["sampling_rate"] = self.sampling_rate



SUMMARY:
**Summary**

The `TextToAudioPipeline` class is a text-to-audio generation pipeline that uses any `AutoModelForTextToWaveform` or `AutoModelForTextToSpectrogram`. It takes an input text and optional other conditional inputs, generates an audio file, and returns it along with its sampling rate.

**Key Implementation Details**

* The pipeline preprocesses the input text using a tokenizer and then passes it to the underlying model for generation.
* If the model is a generative one, additional tensors are added to the `forward_params` dictionary, which are also used for generating the output.
* After obtaining the output from the model, the pipeline uses a vocoder (if available) to convert the spectrogram into a waveform.
* The audio waveform and its sampling rate are then returned along with any other post-processing parameters.

**Notable Dependencies or Requirements**

* `ModelForTextToWaveform` or `ModelForTextToSpectrogram`: These models serve as the underlying logic for generating the text-to-audio output.
* `SpeechT5HifiGan`: This is an optional vocoder that can be used to convert spectrograms into waveforms.

**Interface and Method Signatures**

* `__init__`: Initializes the pipeline with various parameters, including the model and generation configuration.
* `postprocess`: Takes a waveform output from the model, post-processes it if necessary, and returns it along with its sampling rate.


key: chroma:document
string_value:
CODE:

class ZeroShotClassificationArgumentHandler(ArgumentHandler):
    """
    Handles arguments for zero-shot for text classification by turning each possible label into an NLI
    premise/hypothesis pair.
    """

    def _parse_labels(self, labels):
        if isinstance(labels, str):
            labels = [label.strip() for label in labels.split(",") if label.strip()]
        return labels

    def __call__(self, sequences, labels, hypothesis_template):
        if len(labels) == 0 or len(sequences) == 0:
            raise ValueError("You must include at least one label and at least one sequence.")
        if hypothesis_template.format(labels[0]) == hypothesis_template:
            raise ValueError(
                (
                    'The provided hypothesis_template "{}" was not able to be formatted with the target labels. '
                    "Make sure the passed template includes formatting syntax such as {{}} where the label should go."
                ).format(hypothesis_template)
            )

        if isinstance(sequences, str):
            sequences = [sequences]

        sequence_pairs = []
        for sequence in sequences:
            sequence_pairs.extend([[sequence, hypothesis_template.format(label)] for label in labels])



SUMMARY:
**Zero Shot Classification Argument Handler Class Summary**

### Purpose and Functionality

This class handles arguments for zero-shot text classification by generating Natural Language Inference (NLI) premise/hypothesis pairs from provided label sequences. It prepares input data for training zero-shot NLI models.

### Implementation Details

* `_parse_labels` method cleans and normalizes label inputs.
* `__call__` method generates sequence pairs with tailored hypothesis templates.
* **Key Implementation:** Use of formatted hypothesis templates to inject labels into sequence pairs.
 
### Notable Dependencies and Requirements

* Utilizes `ArgumentHandler` base class
* Imports no external dependencies

### Interfaces and Inheritance

Inherited from `ArgumentHandler`.


key: chroma:document
string_value:
CODE:

def is_sagemaker_model_parallel_available():
    # Get the sagemaker specific mp parameters from smp_options variable.
    smp_options = os.getenv("SM_HP_MP_PARAMETERS", "{}")
    try:
        # Parse it and check the field "partitions" is included, it is required for model parallel.
        smp_options = json.loads(smp_options)
        if "partitions" not in smp_options:
            return False
    except json.JSONDecodeError:
        return False

    # Get the sagemaker specific framework parameters from mpi_options variable.
    mpi_options = os.getenv("SM_FRAMEWORK_PARAMS", "{}")
    try:
        # Parse it and check the field "sagemaker_distributed_dataparallel_enabled".
        mpi_options = json.loads(mpi_options)
        if not mpi_options.get("sagemaker_mpi_enabled", False):
            return False
    except json.JSONDecodeError:
        return False
    # Lastly, check if the `smdistributed` module is present.


SUMMARY:
**Summary**

This function checks if SageMaker model parallelism is available and configured correctly. It verifies the presence of specific parameters in environment variables, parsing attempts failing with JSON errors are handled accordingly.

**Key Implementation Details:**

*   Checks for required `partitions` field in `smp_options` json.
*   Verifies configuration of `mpi_options` to enable SageMaker DDP.
*   Ensures availability of the `smdistributed` module using an import check.

**Code Snippets and Dependencies:**

*   Requires parsing and handling environment variable values (`os.getenv()`).
*   Leverages JSON loading for both environment variables (`json.loads()`) and the existence dependency checks.


key: chroma:document
string_value:
CODE:

class SageMakerTrainer(Trainer):
    def __init__(self, args=None, **kwargs):
        warnings.warn(
            "`SageMakerTrainer` is deprecated and will be removed in v5 of Transformers. You can use `Trainer` "
            "instead.",
            FutureWarning,
        )


SUMMARY:
**Class Summary: SageMakerTrainer**

*   **Main Purpose and Functionality:** Depreciated subclass of `Trainer`, intended for replacement by `Trainer` in future versions (v5) of the Transformers library.
*   **Key Implementation Details:**
    *   Expects two parameters: `args` and additional keyword arguments (`**kwargs`) passed to its parent class's `__init__`.
    *   Uses `super().__init__(args=args, **kwargs)` to call the parent class's (non-deprecated) constructor.

**Note:** The `SageMakerTrainer` class is marked as deprecated.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/src/transformers/kernels/falcon_mamba/selective_scan_with_ln_interface.py


key: chroma:document
string_value:
CODE:

class SelectiveScanFn(torch.autograd.Function):
    @staticmethod
    def forward(
        ctx, u, delta, A, B, C, D=None, z=None, delta_bias=None, delta_softplus=False, return_last_state=False
    ):
        if u.stride(-1) != 1:
            u = u.contiguous()
        if delta.stride(-1) != 1:
            delta = delta.contiguous()
        if D is not None:
            D = D.contiguous()
        if B.stride(-1) != 1:
            B = B.contiguous()
        if C.stride(-1) != 1:
            C = C.contiguous()
        if z is not None and z.stride(-1) != 1:
            z = z.contiguous()
        if B.dim() == 3:
            B = rearrange(B, "b dstate l -> b 1 dstate l")
            ctx.squeeze_B = True
        if C.dim() == 3:
            C = rearrange(C, "b dstate l -> b 1 dstate l")
            ctx.squeeze_C = True
        out, x, *rest = selective_scan_cuda.fwd(u, delta, A, B, C, D, z, delta_bias, delta_softplus)
        ctx.delta_softplus = delta_softplus
        ctx.has_z = z is not None
        last_state = x[:, :, -1, 1::2]  # (batch, dim, dstate)
        if not ctx.has_z:
            ctx.save_for_backward(u, delta, A, B, C, D, delta_bias, x)
            return out if not return_last_state else (out, last_state)
        else:
            ctx.save_for_backward(u, delta, A, B, C, D, z, delta_bias, x, out)
            out_z = rest[0]
            return out_z if not return_last_state else (out_z, last_state)

    @staticmethod
    def backward(ctx, dout, *args):
        if not ctx.has_z:
            u, delta, A, B, C, D, delta_bias, x = ctx.saved_tensors
            z = None
            out = None
        else:
            u, delta, A, B, C, D, z, delta_bias, x, out = ctx.saved_tensors
        if dout.stride(-1) != 1:
            dout = dout.contiguous()
        # The kernel supports passing in a pre-allocated dz (e.g., in case we want to fuse the
        # backward of selective_scan_cuda with the backward of chunk).
        # Here we just pass in None and dz will be allocated in the C++ code.
        du, ddelta, dA, dB, dC, dD, ddelta_bias, *rest = selective_scan_cuda.bwd(
            u,
            delta,
            A,
            B,
            C,
            D,
            z,
            delta_bias,
            dout,
            x,
            out,
            None,
            ctx.delta_softplus,
            False,  # option to recompute out_z, not used here
        )
        dz = rest[0] if ctx.has_z else None
        dB = dB.squeeze(1) if getattr(ctx, "squeeze_B", False) else dB
        dC = dC.squeeze(1) if getattr(ctx, "squeeze_C", False) else dC
        return (
            du,
            ddelta,
            dA,
            dB,
            dC,
            dD if D is not None else None,
            dz,
            ddelta_bias if delta_bias is not None else None,
            None,
            None,


SUMMARY:
**Class Summary: SelectiveScanFn**

The `SelectiveScanFn` class implements the PyTorch Autograd function for selective scan, which applies a batched scanning operation to a 4D input tensor.

**Primary Purpose**: Applies batched scan to input tensor using selective scan CUDA kernel.

**Key Implementation Details**:

*   Performs contiguous conversion on tensors `u`, `delta`, and others if their strides don't match.
*   Reshapes tensors `B` and `C` based on their dimensions.
*   Saves variables for backward pass when necessary.
*   Passes inputs to the selective scan CUDA kernel function (`selective_scan_cuda.fwd`) or its backward counterpart (`selective_scan_cuda.bwd`).

**Notable Dependencies and Requirements**:

*   `torch.autograd.Function`
*   `selective_scan_cuda.fwd`
*   `selective_scan_cuda.bwd`

**Interfaces Implemented**:

*   `Forward` function
*   `Backward` function

Note: The summary focuses on the core functionality, key implementation details, and notable dependencies, preserving class, variable, method names consistently.


key: chroma:document
string_value:
CODE:

def rms_norm_forward(
    x,
    weight,
    bias,
    eps=1e-6,
    is_rms_norm=True,
):
    # x (b l) d
    if x.stride(-1) != 1:
        x = x.contiguous()
    weight = weight.contiguous()
    if bias is not None:
        bias = bias.contiguous()
    y = _layer_norm_fwd(x, weight, bias, eps, None, residual_dtype=None, is_rms_norm=is_rms_norm)[0]
    # y (b l) d


SUMMARY:
**Summary:**

The `rms_norm_forward` function normalizes input tensor `x` using RMS Norm. It takes in:
- Input tensor `x`
- Weights and biases for normalization
- Optional parameters `eps` (epsilon value) for numerical stability, default=1e-6, and `is_rms_norm=True`

The function returns the normalized output tensor `y` with shape `(b, l, d)`.

**Key aspects:**

* Normalization using RMS Norm algorithm
* Contiguous tensors to simplify memory access
* Optional use of epsilon for numerical stability

**Notable dependencies and interfaces:**

- `_layer_norm_fwd` function from unknown module (requires further context)
  - Input tensor `x`
  - Weights `weight`
  - Biases (`bias=None`)
  - Data type (`residual_dtype=None`)


key: chroma:document
string_value:
CODE:

def selective_scan_fn(
    u, delta, A, B, C, D=None, z=None, delta_bias=None, delta_softplus=False, return_last_state=False
):
    """if return_last_state is True, returns (out, last_state)
    last_state has shape (batch, dim, dstate). Note that the gradient of the last state is
    not considered in the backward pass.
    """


SUMMARY:
**Function Summary: `selective_scan_fn`**

* **Purpose:** Applies the selective scan function, which processes units `u` using input vectors `A`, `B`, and `C`.
	+ Primary purpose: Perform selective scan with optional delta bias, output softplus function, and last state return flag.

**Key Implementation Details:**

* Applies `SelectiveScanFn.apply` with all input parameters.
* Returns either only the output tensor or the output tensor along with the last state, depending on the `return_last_state` flag.

**Notable Dependencies and Requirements:**

* None explicitly listed; however, `SelectiveScanFn` is assumed to be defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def selective_scan_ref(
    u, delta, A, B, C, D=None, z=None, delta_bias=None, delta_softplus=False, return_last_state=False
):
    """
    u: r(B D L)
    delta: r(B D L)
    A: c(D N) or r(D N)
    B: c(D N) or r(B N L) or r(B N 2L) or r(B G N L) or (B G N L)
    C: c(D N) or r(B N L) or r(B N 2L) or r(B G N L) or (B G N L)
    D: r(D)
    z: r(B D L)
    delta_bias: r(D), fp32

    out: r(B D L)
    last_state (optional): r(B D dstate) or c(B D dstate)
    """
    dtype_in = u.dtype
    u = u.float()
    delta = delta.float()
    if delta_bias is not None:
        delta = delta + delta_bias[..., None].float()
    if delta_softplus:
        delta = F.softplus(delta)
    batch, dim, dstate = u.shape[0], A.shape[0], A.shape[1]
    is_variable_B = B.dim() >= 3
    is_variable_C = C.dim() >= 3
    if A.is_complex():
        if is_variable_B:
            B = torch.view_as_complex(rearrange(B.float(), "... (L two) -> ... L two", two=2))
        if is_variable_C:
            C = torch.view_as_complex(rearrange(C.float(), "... (L two) -> ... L two", two=2))
    else:
        B = B.float()
        C = C.float()
    x = A.new_zeros((batch, dim, dstate))
    ys = []
    deltaA = torch.exp(torch.einsum("bdl,dn->bdln", delta, A))
    if not is_variable_B:
        deltaB_u = torch.einsum("bdl,dn,bdl->bdln", delta, B, u)
    else:
        if B.dim() == 3:
            deltaB_u = torch.einsum("bdl,bnl,bdl->bdln", delta, B, u)
        else:
            B = repeat(B, "B G N L -> B (G H) N L", H=dim // B.shape[1])
            deltaB_u = torch.einsum("bdl,bdnl,bdl->bdln", delta, B, u)
    if is_variable_C and C.dim() == 4:
        C = repeat(C, "B G N L -> B (G H) N L", H=dim // C.shape[1])
    last_state = None
    for i in range(u.shape[2]):
        x = deltaA[:, :, i] * x + deltaB_u[:, :, i]
        if not is_variable_C:
            y = torch.einsum("bdn,dn->bd", x, C)
        else:
            if C.dim() == 3:
                y = torch.einsum("bdn,bn->bd", x, C[:, :, i])
            else:
                y = torch.einsum("bdn,bdn->bd", x, C[:, :, :, i])
        if i == u.shape[2] - 1:
            last_state = x
        if y.is_complex():
            y = y.real * 2
        ys.append(y)
    y = torch.stack(ys, dim=2)  # (batch dim L)
    out = y if D is None else y + u * rearrange(D, "d -> d 1")
    if z is not None:
        out = out * F.silu(z)
    out = out.to(dtype=dtype_in)


SUMMARY:
**Summary:**

The `selective_scan_ref` function performs selective scan operations on input tensors `u`, `delta`, `A`, `B`, and `C`. It applies the following key steps:

1.  **Softplus and Bias Correction**: Applies softplus activation to `delta` if enabled (`delta_softplus=True`). Adds a bias term (`delta_bias`) to `delta` if provided.
2.  **Data Type Conversion**: Converts input tensors and biases to floating-point type (`float()`) for operations.
3.  **Variable Dimension Handling**: Handles variable dimension inputs by rearranging or repeating tensor views accordingly.

Key Implementation Details:

*   Computes intermediate results, such as `deltaA` using the exponential function and matrix multiplication.
*   Applies complex-to-real conversion if necessary, using `torch.view_as_complex` and `rearrange`.
*   Iterates through the last dimension of `u`, accumulating results in variable-size lists (`ys`) to construct the final output tensor.

Notable Dependencies and Requirements:

*   **Tensorflow/PyTorch library functions**, such as `F.softplus`, `torch.view_as_complex`, and `rearrange`.
*   **External dependencies**: The code relies on custom implementation of the `silu` activation function, which is used in the final output computation.


key: chroma:document
string_value:
CODE:

class MambaInnerFn(torch.autograd.Function):
    @staticmethod
    @custom_fwd
    def forward(
        ctx,
        xz,
        conv1d_weight,
        conv1d_bias,
        x_proj_weight,
        delta_proj_weight,
        out_proj_weight,
        out_proj_bias,
        A,
        B=None,
        C=None,
        D=None,
        delta_bias=None,
        B_proj_bias=None,
        C_proj_bias=None,
        delta_softplus=True,
        checkpoint_lvl=1,
        b_rms_weight=None,
        c_rms_weight=None,
        dt_rms_weight=None,
        b_c_dt_rms_eps=1e-6,
    ):
        """
        xz: (batch, dim, seqlen)
        """
        assert causal_conv1d_cuda is not None, "causal_conv1d_cuda is not available. Please install causal-conv1d."
        assert checkpoint_lvl in [0, 1]
        L = xz.shape[-1]
        delta_rank = delta_proj_weight.shape[1]
        d_state = A.shape[-1] * (1 if not A.is_complex() else 2)
        if torch.is_autocast_enabled():
            x_proj_weight = x_proj_weight.to(dtype=torch.get_autocast_gpu_dtype())
            delta_proj_weight = delta_proj_weight.to(dtype=torch.get_autocast_gpu_dtype())
            out_proj_weight = out_proj_weight.to(dtype=torch.get_autocast_gpu_dtype())
            out_proj_bias = (
                out_proj_bias.to(dtype=torch.get_autocast_gpu_dtype()) if out_proj_bias is not None else None
            )
        if xz.stride(-1) != 1:
            xz = xz.contiguous()
        conv1d_weight = rearrange(conv1d_weight, "d 1 w -> d w")
        x, z = xz.chunk(2, dim=1)
        conv1d_bias = conv1d_bias.contiguous() if conv1d_bias is not None else None
        conv1d_out = causal_conv1d_cuda.causal_conv1d_fwd(x, conv1d_weight, conv1d_bias, None, None, None, True)
        # We're being very careful here about the layout, to avoid extra transposes.
        # We want delta to have d as the slowest moving dimension
        # and L as the fastest moving dimension, since those are what the ssm_scan kernel expects.
        x_dbl = F.linear(rearrange(conv1d_out, "b d l -> (b l) d"), x_proj_weight)  # (bl d)
        delta = rearrange(delta_proj_weight @ x_dbl[:, :delta_rank].t(), "d (b l) -> b d l", l=L)
        ctx.is_variable_B = B is None
        ctx.is_variable_C = C is None
        ctx.B_proj_bias_is_None = B_proj_bias is None
        ctx.C_proj_bias_is_None = C_proj_bias is None
        if B is None:  # variable B
            B = x_dbl[:, delta_rank : delta_rank + d_state]  # (bl dstate)
            if B_proj_bias is not None:
                B = B + B_proj_bias.to(dtype=B.dtype)
            if not A.is_complex():
                # B = rearrange(B, "(b l) dstate -> b dstate l", l=L).contiguous()
                B = rearrange(B, "(b l) dstate -> b 1 dstate l", l=L).contiguous()
            else:
                B = rearrange(B, "(b l) (dstate two) -> b 1 dstate (l two)", l=L, two=2).contiguous()
        else:
            if B.stride(-1) != 1:
                B = B.contiguous()
        if C is None:  # variable C
            C = x_dbl[:, -d_state:]  # (bl dstate)
            if C_proj_bias is not None:
                C = C + C_proj_bias.to(dtype=C.dtype)
            if not A.is_complex():
                # C = rearrange(C, "(b l) dstate -> b dstate l", l=L).contiguous()
                C = rearrange(C, "(b l) dstate -> b 1 dstate l", l=L).contiguous()
            else:
                C = rearrange(C, "(b l) (dstate two) -> b 1 dstate (l two)", l=L, two=2).contiguous()
        else:
            if C.stride(-1) != 1:
                C = C.contiguous()
        if D is not None:
            D = D.contiguous()

        if b_rms_weight is not None:
            B = rearrange(B, "b 1 dstate l -> (b l) dstate", l=L).contiguous()
            B = rms_norm_forward(B, b_rms_weight, bias=None, eps=b_c_dt_rms_eps)
            B = rearrange(B, "(b l) dstate -> b 1 dstate l", l=L).contiguous()
        if c_rms_weight is not None:
            C = rearrange(C, "b 1 dstate l -> (b l) dstate", l=L).contiguous()
            C = rms_norm_forward(C, c_rms_weight, bias=None, eps=b_c_dt_rms_eps)
            C = rearrange(C, "(b l) dstate -> b 1 dstate l", l=L).contiguous()
        if dt_rms_weight is not None:
            delta = rearrange(delta, "b d l -> (b l) d", l=L).contiguous()
            delta = rms_norm_forward(delta, dt_rms_weight, bias=None, eps=b_c_dt_rms_eps)
            delta = rearrange(delta, "(b l) d -> b d l", l=L).contiguous()

        out, scan_intermediates, out_z = selective_scan_cuda.fwd(
            conv1d_out, delta, A, B, C, D, z, delta_bias, delta_softplus
        )
        ctx.delta_softplus = delta_softplus
        ctx.out_proj_bias_is_None = out_proj_bias is None
        ctx.checkpoint_lvl = checkpoint_lvl
        ctx.b_rms_weight = b_rms_weight
        ctx.c_rms_weight = c_rms_weight
        ctx.dt_rms_weight = dt_rms_weight
        ctx.b_c_dt_rms_eps = b_c_dt_rms_eps
        if checkpoint_lvl >= 1:  # Will recompute conv1d_out and delta in the backward pass
            conv1d_out, delta = None, None
        ctx.save_for_backward(
            xz,
            conv1d_weight,
            conv1d_bias,
            x_dbl,
            x_proj_weight,
            delta_proj_weight,
            out_proj_weight,
            conv1d_out,
            delta,
            A,
            B,
            C,
            D,
            delta_bias,
            scan_intermediates,
            b_rms_weight,
            c_rms_weight,
            dt_rms_weight,
            out,
        )
        return F.linear(rearrange(out_z, "b d l -> b l d"), out_proj_weight, out_proj_bias)

    @staticmethod
    @custom_bwd
    def backward(ctx, dout):
        # dout: (batch, seqlen, dim)
        assert causal_conv1d_cuda is not None, "causal_conv1d_cuda is not available. Please install causal-conv1d."
        (
            xz,
            conv1d_weight,
            conv1d_bias,
            x_dbl,
            x_proj_weight,
            delta_proj_weight,
            out_proj_weight,
            conv1d_out,
            delta,
            A,
            B,
            C,
            D,
            delta_bias,
            scan_intermediates,
            b_rms_weight,
            c_rms_weight,
            dt_rms_weight,
            out,
        ) = ctx.saved_tensors
        L = xz.shape[-1]
        delta_rank = delta_proj_weight.shape[1]
        d_state = A.shape[-1] * (1 if not A.is_complex() else 2)
        x, z = xz.chunk(2, dim=1)
        if dout.stride(-1) != 1:
            dout = dout.contiguous()
        if ctx.checkpoint_lvl == 1:
            conv1d_out = causal_conv1d_cuda.causal_conv1d_fwd(x, conv1d_weight, conv1d_bias, None, None, None, True)
            delta = rearrange(delta_proj_weight @ x_dbl[:, :delta_rank].t(), "d (b l) -> b d l", l=L)
            if dt_rms_weight is not None:
                delta = rearrange(delta, "b d l -> (b l) d", l=L).contiguous()
                delta = rms_norm_forward(delta, ctx.dt_rms_weight, None, ctx.b_c_dt_rms_eps)
                delta = rearrange(delta, "(b l) d -> b d l", l=L).contiguous()
            if b_rms_weight is not None:
                # Recompute & RMSNorm B
                B = rearrange(B, "b 1 dstate l -> (b l) dstate", l=L).contiguous()
                B = rms_norm_forward(B, ctx.b_rms_weight, None, ctx.b_c_dt_rms_eps)
                B = rearrange(B, "(b l) dstate -> b 1 dstate l", l=L).contiguous()
            if c_rms_weight is not None:
                # Recompute & RMSNorm C
                C = rearrange(C, "b 1 dstate l -> (b l) dstate", l=L).contiguous()
                C = rms_norm_forward(C, ctx.c_rms_weight, None, ctx.b_c_dt_rms_eps)
                C = rearrange(C, "(b l) dstate -> b 1 dstate l", l=L).contiguous()

        # The kernel supports passing in a pre-allocated dz (e.g., in case we want to fuse the
        # backward of selective_scan_cuda with the backward of chunk).
        dxz = torch.empty_like(xz)  # (batch, dim, seqlen)
        dx, dz = dxz.chunk(2, dim=1)
        dout = rearrange(dout, "b l e -> e (b l)")
        dout_y = rearrange(out_proj_weight.t() @ dout, "d (b l) -> b d l", l=L)
        dconv1d_out, ddelta, dA, dB, dC, dD, ddelta_bias, dz, out_z = selective_scan_cuda.bwd(
            conv1d_out,
            delta,
            A,
            B,
            C,
            D,
            z,
            delta_bias,
            dout_y,
            scan_intermediates,
            out,
            dz,
            ctx.delta_softplus,
            True,  # option to recompute out_z
        )
        dout_proj_weight = torch.einsum("eB,dB->ed", dout, rearrange(out_z, "b d l -> d (b l)"))
        dout_proj_bias = dout.sum(dim=(0, 1)) if not ctx.out_proj_bias_is_None else None
        dD = dD if D is not None else None
        dx_dbl = torch.empty_like(x_dbl)
        dB_proj_bias = None
        if ctx.is_variable_B:
            if not A.is_complex():
                dB = rearrange(dB, "b 1 dstate l -> (b l) dstate").contiguous()
            else:
                dB = rearrange(dB, "b 1 dstate (l two) -> (b l) (dstate two)", two=2).contiguous()
            dB_proj_bias = dB.sum(0) if not ctx.B_proj_bias_is_None else None
            dx_dbl[:, delta_rank : delta_rank + d_state] = dB  # (bl d)
            dB = None
        dC_proj_bias = None
        if ctx.is_variable_C:
            if not A.is_complex():
                dC = rearrange(dC, "b 1 dstate l -> (b l) dstate").contiguous()
            else:
                dC = rearrange(dC, "b 1 dstate (l two) -> (b l) (dstate two)", two=2).contiguous()
            dC_proj_bias = dC.sum(0) if not ctx.C_proj_bias_is_None else None
            dx_dbl[:, -d_state:] = dC  # (bl d)
            dC = None
        ddelta = rearrange(ddelta, "b d l -> d (b l)")
        ddelta_proj_weight = torch.einsum("dB,Br->dr", ddelta, x_dbl[:, :delta_rank])
        dx_dbl[:, :delta_rank] = torch.einsum("dB,dr->Br", ddelta, delta_proj_weight)
        dconv1d_out = rearrange(dconv1d_out, "b d l -> d (b l)")
        dx_proj_weight = torch.einsum("Br,Bd->rd", dx_dbl, rearrange(conv1d_out, "b d l -> (b l) d"))
        dconv1d_out = torch.addmm(dconv1d_out, x_proj_weight.t(), dx_dbl.t(), out=dconv1d_out)
        dconv1d_out = rearrange(dconv1d_out, "d (b l) -> b d l", b=x.shape[0], l=x.shape[-1])
        # The kernel supports passing in a pre-allocated dx (e.g., in case we want to fuse the
        # backward of conv1d with the backward of chunk).
        dx, dconv1d_weight, dconv1d_bias, *_ = causal_conv1d_cuda.causal_conv1d_bwd(
            x, conv1d_weight, conv1d_bias, dconv1d_out, None, None, None, dx, False, True
        )
        dconv1d_bias = dconv1d_bias if conv1d_bias is not None else None
        dconv1d_weight = rearrange(dconv1d_weight, "d w -> d 1 w")
        return (
            dxz,
            dconv1d_weight,
            dconv1d_bias,
            dx_proj_weight,
            ddelta_proj_weight,
            dout_proj_weight,
            dout_proj_bias,
            dA,
            dB,
            dC,
            dD,
            ddelta_bias if delta_bias is not None else None,
            # 6-None are delta_softplus, checkpoint_lvl, b_rms_weight, c_rms_weight, dt_rms_weight, b_c_dt_rms_eps
            dB_proj_bias,
            dC_proj_bias,
            None,
            None,
            None,
            None,
            None,
            None,


SUMMARY:
This is the implementation of the backward pass for the `causal_conv1d` module in PyTorch. The `causal_conv1d_bwd` function implements the standard convolutional neural network (CNN) backward pass, with the additional modification that it handles causal convolutions.

The overall process can be summarized as follows:

1. Initialize variables: Calculate intermediate tensors and variables such as `dx`, `ddelta` and `convolutions`.
2. Compute convolution results: Compute the convolution of `x` and `w_out` to produce `conv_v`.
3. Backpropagate activations with bias: For each position along `b_l`, calculate `zout[bl] = aout[b][l] * weights[l bl].bias`. Initialize this value for each new position.
4. Compute biases through the backward pass (BWD) process and obtain values of B and c.
5. Update weights (x2)
6. For `ddelta` compute
    1. `dx_dbl[:, bl - d_state, l + b] = delta_proj_weight[bl - dState, l + b]`, which will be equal to the first output element after computing biases through BWD and setting the bias values of delta.
7. For weights `Dout[]`, compute weight updates based on
    1. `ddelta_B[l][bl - ddelta_state - (ddelta_state-1)]`
8. Backpropagate `x` through W `to` b with the new gradients for weights and biases by computing `dx_out[l + bi] = wout[m * in_channels, bl][l+i] * d convout[bm][i]`, finally calculate `conv_v [bl + i]`.
9. The variables such as `dz`, `ddelta_bias` are updated with the results of each computation.
10. The final result is to update weights and biases.

This backward pass takes into account the causal nature of the convolution, which means it only considers the most recent input value when computing the output. This has implications for how the derivative of the output is propagated backwards through the network.


key: chroma:document
string_value:
CODE:

def mamba_inner_fn(
    xz,
    conv1d_weight,
    conv1d_bias,
    x_proj_weight,
    delta_proj_weight,
    out_proj_weight,
    out_proj_bias,
    A,
    B=None,
    C=None,
    D=None,
    delta_bias=None,
    B_proj_bias=None,
    C_proj_bias=None,
    delta_softplus=True,
    checkpoint_lvl=1,
    b_rms_weight=None,
    c_rms_weight=None,
    dt_rms_weight=None,
    b_c_dt_rms_eps=1e-6,
):
    return MambaInnerFn.apply(
        xz,
        conv1d_weight,
        conv1d_bias,
        x_proj_weight,
        delta_proj_weight,
        out_proj_weight,
        out_proj_bias,
        A,
        B,
        C,
        D,
        delta_bias,
        B_proj_bias,
        C_proj_bias,
        delta_softplus,
        checkpoint_lvl,
        b_rms_weight,
        c_rms_weight,
        dt_rms_weight,
        b_c_dt_rms_eps,


SUMMARY:
**Concise Summary**

The `mamba_inner_fn` function is an application of the `MambaInnerFn.apply` method. It takes in and returns various inputs, including weights, biases, activation functions, and data types:

*   Main purpose: Applies a complex mathematical operation involving convolutional layers, projections, and transformations.
*   Key implementation details: Utilizes `conv1d_weight`, `x_proj_weight`, `out_proj_weight` for weighted convolutions; uses `MambaInnerFn.apply` to integrate multiple components of the Mamba algorithm.

Notable dependencies:
- Uses external module 'activation_modules.py' (implied, but not shown)

Required inputs:
*   Data types: `xz`, `conv1d_weight`, `conv1d_bias`, `x_proj_weight`, `delta_proj_weight`, `out_proj_weight`, `out_proj_bias`
*   External dependencies are assumed to be part of a larger system or module not explicitly defined in this code snippet.


key: chroma:document
string_value:
CODE:
### Fake tools for test
def classifier(text, labels):


SUMMARY:
**Code Summary**

### Function Overview

`classifier` takes in `text` and `labels` as input and returns a formatted string with a classification message.

### Key Implementation Details

- The function uses an f-string to create the output string, which allows for dynamic formatting of the input data.
- No complex algorithms or patterns are employed; instead, the function simply concatenates static strings with input data.

### Dependencies and Requirements

None explicit dependencies listed. However, it assumes:

* `text` and `labels` are valid input parameters (type not specified).
* An understanding of Python's f-string formatting feature is assumed.
* No external libraries or resources are required for its execution.


key: chroma:document
string_value:
CODE:

def translator(text, src_lang, tgt_lang):


SUMMARY:
**Code Summary**

### Function Purpose

`translator()` is a simple language translation function that generates a literal translation message.

**Key Implementation Details**

* The `translator()` function takes three parameters: `text`, `src_lang`, and `tgt_lang`.
* It returns a string in the format "This is the translation of < text > from < src_lang > to < tgt_lang >."

### Notable Dependencies or Requirements

None explicitly listed, but assumes Python 3.6+ for f-string formatting.

**Notes**

The function appears to be a placeholder or demonstration code, lacking any actual translation logic.


key: chroma:document
string_value:
CODE:

def speaker(text):


SUMMARY:
**Summary: Speaker Function**

The `speaker` function takes a text input and returns a string with a specific format indicating it's a rewritten speech read as text (`f"""This is actually a sound reading {text}."""`). This transformation appears to convert text dialogue into spoken audio representation for presentation purposes. 

**Key Implementation Details:**

- Replaces placeholders (`{text}`) in the template with actual input text.
- Uses an f-string format to embed and manipulate the speech read data.

**Notable Dependencies or Requirements:**

- None mentioned, suggesting a self-contained implementation focused on speech-to-text rewriting.


key: chroma:document
string_value:
CODE:

def transcriber(audio):
    if "sound" not in audio:
        raise ValueError(f"`audio` ({audio}) is not a sound.")


SUMMARY:
**Summary**

Transcriber Function Summary
==========================

### Purpose and Functionality

The `transcriber` function takes an `audio` object as input, verifies that it represents a sound using the presence of the string `"sound"`, and raises a `ValueError` if invalid. If `audio` is valid, it returns a descriptive text message transcribing the audio.

### Key Implementation Details

* Verifies audio validity by checking for presence of `"sound"` string.
* Raises `ValueError` with error message indicating invalid audio input.

### Dependencies/Requirements

None explicitly stated; assumes `ValueError` exception and string concatenation (`f-strings`) functionality are available.


key: chroma:document
string_value:
CODE:

def image_generator(prompt):


SUMMARY:
**Summary:**

The `image_generator` function takes a `prompt` as input and returns a string that represents an image based on the provided prompt.

**Key Details:**

* Function signature: `def image_generator(prompt):`
* Returns a string representation of an image.

**Implementation Notes:**

None. The implementation is simple and returns a fixed string without any images.

Note: This summary assumes that the actual function may be overridden or extended in other parts of the codebase.


key: chroma:document
string_value:
CODE:

def image_captioner(image):
    if "image" not in image:
        raise ValueError(f"`image` ({image}) is not an image.")


SUMMARY:
**Summary:**

This function, `image_captioner`, takes an `image` object as input and returns a string caption describing it. The main purpose is to validate the input `image` and raise a `ValueError` if it's not a valid image.

**Key Implementation Details:**

* A simple validation check using `if "image" in image` (exact condition)
* Raises a `ValueError` with a custom error message using f-string formatting

**Notable Dependencies or Requirements:**

* None explicitly mentioned
* Presumably relies on the Python interpreter and basic standard library data types.


key: chroma:document
string_value:
CODE:

def image_transformer(image, prompt):
    if "image" not in image:
        raise ValueError(f"`image` ({image}) is not an image.")


SUMMARY:
**Code Summary**

### Functionality

The `image_transformer` function transforms a given `image` by generating a metadata string based on the provided `prompt`.

* Primary purpose: Generates a metadata string with image information and transformation prompt.

### Implementation Details

* No significant algorithms or patterns detected.

### Dependencies and Requirements

 None. 

Note: The `image` variable is expected to be an object that has its own structure, which is not explicitly handled in this function.


key: chroma:document
string_value:
CODE:

def question_answerer(text, question):


SUMMARY:
**Summary:**

A simple Python function named `question_answerer` that takes two strings as input, `text` and `question`. The function returns a constructed string in response to the provided question.

Key details:

* **Function Signature:** `question_answerer(text: str, question: str)`
* **Primary Purpose:** Generates an answer based on the input text and question.
* **Implementation Details:**
	+ Uses f-string formatting to construct the response.
* **Dependencies:** None.
* **Interfaces:** Not part of this function.

**RAG System Summary:**
`function question_answerer(text, question): string`
` purpose: generate response to provided question `
` details: uses f-strings for construction`
` dependencies: none `


key: chroma:document
string_value:
CODE:

def image_qa(image, question):
    if "image" not in image:
        raise ValueError(f"`image` ({image}) is not an image.")


SUMMARY:
**Summary:**

`image_qa` Function:

* Purpose: Verifies if `image` meets basic requirements and returns a response based on `question`.
* Primary function: `is_image` verification followed by a concatenation-based answer generation.

Key Implementation detail: The core logic is contained within the simple conditional check and string concatenation operations.

No notable dependencies or requirements are mentioned.


key: chroma:document
string_value:
CODE:

def text_downloader(url):


SUMMARY:
**Summary**

The `text_downloader` function takes a URL as input and returns a string containing the provided website's title (not directly accessible in this implementation). The primary purpose is to retrieve and format a generic text description for any given URL.

**Key Implementation Details:**

* Uses an f-string to concatenate HTML title tag content (not actually accessing it)
	+ Method signature: `return f"This is the content of {url}."`

**Notable Dependencies or Requirements:**

* None explicitly referenced in this snippet
* No external dependencies listed, but note that a typical text downloader might rely on web scraping libraries to access website titles.


key: chroma:document
string_value:
CODE:

def summarizer(text):


SUMMARY:
**Summary:**

The `summarizer` function takes in a `text` parameter and returns a formatted string summarizing the input text.

* **Purpose:** Provides a basic summary functionality for the given text.
* **Implementation:** Returns a fixed string using f-strings, with the provided text inserted into the summary template.

**Additional Details:**

* The implementation details are simplified, lacking any actual summarization logic.


key: chroma:document
string_value:
CODE:

def video_generator(prompt, seconds=2):


SUMMARY:
**Summary:**

The `video_generator` function takes two parameters:

* `prompt`: a string to be used as the content of the generated video
* `seconds`: an optional parameter (default 2) that specifies the duration of the video

It returns a formatted string indicating the creation of a video with the provided `prompt`.

**Key Implementation Details:**

* Returns a simple, pre-formatted string using f-string formatting.

**Notable Dependencies or Requirements:**

None noted. It is a standalone function without any external dependencies.

**Class/Function Name Summary:**

`video_generator`: A lightweight function to generate a video description based on input `prompt`.


key: chroma:document
string_value:
CODE:

def document_qa(image, question):


SUMMARY:
**Summary**

### Function Name: `document_qa`

**Purpose:** Generates a string that answers a QA question about a document.


*Input Parameters:* 
`image`: The document being queried (`string`)
`question`: The relevant question (`string`)


**Implementation Details:**

* The function uses an f-string to format the answer response.
* No calculations or data processing are performed; the answer is simply a generated string.


### Dependencies

None, using only Python's built-in data types and functions.


key: chroma:document
string_value:
CODE:

def image_segmenter(image, prompt):


SUMMARY:
**Summary**

This function, `image_segmenter`, takes an `image` and a `prompt` as input and returns a string describing a segmentation result. The function simply concatenates the prompt and image information into a formatted string.

**Detailed Analysis**

* **Functionality**: Generates a descriptive text based on an input image and prompt.
* **Key Implementation Detail**: A simple string formatting using f-strings, where `image` and `prompt` are used to create a concatenated string.
* **Notable Dependencies/Requirements**: None.
* **Interface**: Returns a string (`str`).

**RAG System Representation**

`image_segmenter(image: str, prompt: str) -> str:`
Generates a descriptive text based on an input image and prompt.


key: chroma:document
string_value:
CODE:

class Problem:
    """
    A class regrouping all the information to solve a problem on which we will evaluate agents.

    Args:
        task (`str` ou `list[str]`):
            One or several descriptions of the task to perform. If a list, it should contain variations on the
            phrasing, but for the same task.
        inputs (`list[str]` or `dict[str, str]`):
            The inputs that will be fed to the tools. For this testing environment, only strings are accepted as
            values. Pass along a dictionary when you want to specify the values of each inputs, or just the list of
            inputs expected (the value used will be `<<input_name>>` in this case).
        answer (`str` or `list[str]`):
            The theoretical answer (or list of possible valid answers) to the problem, as code.
    """

    def __init__(self, task, inputs, answer):
        self.task = task
        self.inputs = inputs


SUMMARY:
**Class Summary: Problem**

The `Problem` class encapsulates information to solve a problem, composed of:

*   **Initialization Parameters:** 
    *   `task`: a string or list of strings describing the task (e.g., "move red ball").
    *   `inputs`: either a list of strings or dictionary to specify input values. If using strings, it's assumed that `input_name` will be used in actual testing.
    *   `answer`: a string representing the theoretical answer or a list of possible valid answers.
*   **Properties:** 
    *   `task`
    *   `inputs` (as dictionary if original input was list)
    *   `answer`

**Key Implementation:**

The class uses a dictionary-like structure (`self.inputs`) to store inputs. If the input is provided as a list, it assumes that the string representation of each variable will be used.

No explicit documentation or additional implementation details were found in this snippet.


key: chroma:document
string_value:
CODE:

def get_theoretical_tools(agent_answer, theoretical_answer, code_answer):
    if not isinstance(theoretical_answer, list):
        return {name for name in TEST_TOOLS if name in code_answer}

    if isinstance(agent_answer, dict):
        for one_answer, one_code in zip(theoretical_answer, code_answer):
            if one_answer in agent_answer.values():
                return {name for name in TEST_TOOLS if name in one_code}

    for one_answer, one_code in zip(theoretical_answer, code_answer):
        if agent_answer == one_answer:
            return {name for name in TEST_TOOLS if name in one_code}



SUMMARY:
**Summary:**

The `get_theoretical_tools` function filters matching tools from the `TEST_TOOLS` list based on three input parameters: agent answers, theoretical answers, and code answers. It returns a set of tool names that correspond to the overlapping tools in either the `CODE_ANSWER` or `THEORETICAL_ANSWER`.

**Main Purpose and Functionality:**
This function takes into account different types of inputs (list vs dict) and compares them with `agent_answer`, returning matching tools.

**Key Implementation Details:**

* Uses set comprehension to filter tool names
* Compares `agent_answer` with individual elements in `THEORETICAL_ANSWER` or `CODE_ANSWER`
* Uses the `zip` function to iterate over both lists simultaneously

**Notable Dependencies or Requirements:**

* `TEST_TOOLS`: a predefined list of tool names (not shown in this code snippet)
* `CODE_ANSWER` and `THEORETICAL_ANSWER`: input parameters with potential differing content formats


key: chroma:document
string_value:
CODE:

def evaluate_code(code, inputs=None, state=None, verbose=False, return_interpretor_error=False):
    tools = BASE_PYTHON_TOOLS.copy()
    for name, tool in TEST_TOOLS.items():
        if name not in code:
            continue
        tools[name] = tool

    if isinstance(inputs, dict):
        inputs = inputs.copy()
    elif inputs is not None:
        inputs = {inp: f"<<{inp}>>" for inp in inputs}

    if state is not None:
        state.update(inputs)
    else:
        state = inputs

    try:
        return evaluate(code, tools, state)
    except InterpreterError as e:
        return str(e)
    except Exception as e:
        if verbose:
            print(e)


SUMMARY:
**Function Summary: `evaluate_code`**

### Purpose and Functionality
Evaluates a code snippet with given inputs, using predefined Python tools, and returns the result.

#### Key Implementation Details

- Evaluates provided `code` with customized `inputs`, utilizing predefined `tools`.
- Updates input dictionary (`state`) as needed.
- Calls recursive `evaluate` function to execute code.

#### Notable Dependencies and Requirements

*   Uses `BASE_PYTHON_TOOLS` and `TEST_TOOLS` classes/data structures from another module.
*   Imports the `InterpreterError` exception from an external module (not shown in this example).
*   Utilizes the `print` function for logging output (dependent on verbose mode).

### Method Signatures

- `evaluate_code(code, inputs=None, state=None, verbose=False, return_interpretor_error=False)`

        This function call has parameters:
    -  `code`: the code snippet to be evaluated.
    -  `inputs`: dictionary of input values; can also be a list or string.
    -  `state`: dictionary of additional inputs (defaults to None).
    -  `verbose`: boolean toggle for logging exceptions (default: False).
    -  `return_interpretor_error`: boolean return type specification (None by default).

### Interfaces Inherited Implementations

The function utilizes the following external modules/interfaces:

-   `BASE_PYTHON_TOOLS`
-   `TEST_TOOLS` 
-   `InterpreterError`


key: chroma:document
string_value:
CODE:

def score_code(agent_answer, theoretical_answer, verbose: bool = False):
    if verbose:
        print(agent_answer, theoretical_answer)
    theoretical_answer = theoretical_answer if isinstance(theoretical_answer, list) else [theoretical_answer]

    if agent_answer in theoretical_answer:
        if verbose:
            print("Perfect!")
        return 1
    elif isinstance(agent_answer, dict) and any(v in theoretical_answer for v in agent_answer.values()):
        if verbose:
            print("Almsot perfect, result in state!")
        return 0.75
    else:
        if verbose:
            print("Result is not the right one but code executed.")


SUMMARY:
**Concise Summary:**

```python
def score_code(agent_answer, theoretical_answer, verbose: bool = False):
    # Verifies and prints input values (if verbose)    
    scores = {  # Map to return values 
                1 if agent_answer in theoretical_answer else
            0.75 if any(v in theoretical_answer for v in agent_answer.values()) else None }
    return scores[1]
```

**Highlighted Details:**

* Returns score based on `agent_answer` comparison with `theoretical_answer`
* Uses dictionary mapping to store and retrieve scores
* Verifies input values using `if-else` statements and prints messages when `verbose` is True

**Notable Dependencies/Requirements:**

* None specified in this code snippet


key: chroma:document
string_value:
CODE:

def evaluate_one_result(code, agent_answer, theoretical_answer, answer, verbose=False):
    tools_in_code = {name for name in TEST_TOOLS if f"`{name}`" in code}
    theoretical_tools = get_theoretical_tools(agent_answer, theoretical_answer, answer)
    if tools_in_code == theoretical_tools:
        tool_selection_score = 1.0
        tool_selection_errors = None
    else:
        missing_tools = len(theoretical_tools - tools_in_code)
        unexpected_tools = len(tools_in_code - theoretical_tools)
        tool_selection_score = max(0, 1.0 - 0.25 * missing_tools - 0.25 * unexpected_tools)

        tool_selection_errors = {
            "selected_tools": tools_in_code,
            "theoretical_tools": theoretical_tools,
        }

    tools_in_code = {name for name in TEST_TOOLS if name in code}
    if tools_in_code == theoretical_tools:
        tool_used_score = 1.0
        tool_used_errors = None
    else:
        missing_tools = len(theoretical_tools - tools_in_code)
        unexpected_tools = len(tools_in_code - theoretical_tools)
        tool_used_score = max(0, 1.0 - 0.25 * missing_tools - 0.25 * unexpected_tools)

        tool_used_errors = {
            "selected_tools": tools_in_code,
            "theoretical_tools": theoretical_tools,
        }

    score = score_code(agent_answer, theoretical_answer, verbose=verbose)
    if score < 1.0:
        code_errors = {
            "code_produced": code,
            "evaluation": agent_answer,
            "theoretical_answer": theoretical_answer,
        }
    else:
        code_errors = None



SUMMARY:
**Function Summary: `evaluate_one_result`**

**Purpose:** Evaluate the similarity between a provided code and theoretical tools, with agent-generated answer, to calculate scores for tool selection and usage.

**Main Functionality:**

1. **Tool Selection**: Compare `tools_in_code` from the provided code to `theoretical_tools` using set operations.
2. **Calculate Tool Scores**: Compute scores (`tool_selection_score` & `tool_used_score`) based on matching and mismatching tools, ranging from 0 (no match) to 1 (full match).
3. **Code Evaluation**: Use `score_code` function to evaluate the overall code quality.
4. **Summary Results**: Return a tuple containing:
	* Scores (`tool_selection_score`, `tool_used_score`, `score`)
	* Errors details for tool selection and usage
	* Code errors if score is below threshold (0.1)

**Key Implementation Details:**

* `tools_in_code` comparison using set intersection and union.
* Calculation of missing and unexpected tools based on set difference.
* Use of scoring functions (`max`, `str.format`) to balance matching and mismatching tool weights.

**Dependencies and Requirements:**

* `get_theoretical_tools`: External function to retrieve theoretical tools from agent answer, theoretical answer, and answer.
* `score_code`: External function to evaluate code quality based on agent answer, theoretical answer.
* `TEST_TOOLS`: Dictionary of test tools used in the evaluation.


key: chroma:document
string_value:
CODE:

def evaluate_agent(agent, batch_size=8, verbose=False, return_errors=False):
    """
    Evaluates a new agent on all `EVALUATION_TASKS`.

    Example:

    ```py
    agent = NewOpenAiAgent(model="text-davinci-003", api_key=your_api_key)
    bads = new_evaluate_agent(agent)
    for bad in bads:
        print(bad)
    ```
    """
    # Sanity check
    agent_tools = set(agent.toolbox.keys())
    if agent_tools != set(TEST_TOOLS):
        missing_tools = set(TEST_TOOLS) - agent_tools
        unexpected_tools = set(agent_tools) - TEST_TOOLS
        raise ValueError(
            f"Fix the test tools in the evaluate_agent module. Tools mising: {missing_tools}. Extra tools: {unexpected_tools}."
        )

    eval_tasks = []
    eval_idx = []
    for idx, pb in enumerate(EVALUATION_TASKS):
        if isinstance(pb.task, list):
            eval_tasks.extend(pb.task)
            eval_idx.extend([idx] * len(pb.task))
        else:
            eval_tasks.append(pb.task)
            eval_idx.append(idx)

    tool_selection_score = 0
    tool_used_score = 0
    code_score = 0

    if return_errors:
        tool_selection_errors = {}
        tool_used_errors = {}
        code_errors = {}

    for start_idx in range(0, len(eval_tasks), batch_size):
        end_idx = min(start_idx + batch_size, len(eval_tasks))
        batch_tasks = eval_tasks[start_idx:end_idx]

        results = [agent.run(task, return_generated_code=True) for task in batch_tasks]

        for idx, result in enumerate(results):
            problem = EVALUATION_TASKS[eval_idx[start_idx + idx]]
            if verbose:
                print(f"====Task {start_idx + idx}====\n{batch_tasks[idx]}\n")
            code = agent.extract_action(result, split_token="Answer:")

            # Evaluate agent answer and code answer
            agent_answer = evaluate_code(code, problem.inputs, verbose=verbose)
            if isinstance(problem.answer, list):
                theoretical_answer = [evaluate_code(answer, problem.inputs) for answer in problem.answer]
            else:
                theoretical_answer = evaluate_code(problem.answer, problem.inputs)

            scores, errors = evaluate_one_result(
                code, agent_answer, theoretical_answer, problem.answer, verbose=verbose
            )

            tool_selection_score += scores[0]
            tool_used_score += scores[1]
            code_score += scores[2]

            if return_errors:
                if errors[0] is not None:
                    tool_selection_errors[batch_tasks[idx]] = errors[0]
                if errors[1] is not None:
                    tool_used_errors[batch_tasks[idx]] = errors[1]
                if errors[2] is not None:
                    code_errors[batch_tasks[idx]] = errors[2]

    scores = {
        "tool selection score": 100 * (tool_selection_score / len(eval_tasks)),
        "tool used score": 100 * (tool_used_score / len(eval_tasks)),
        "code score": 100 * (code_score / len(eval_tasks)),
    }

    if return_errors:
        return scores, tool_selection_errors, tool_used_errors, code_errors
    else:


SUMMARY:
**Summary**

The `evaluate_agent` function evaluates an agent on all tasks in `EVALUATION_TASKS`. It assesses the agent's performance by measuring its ability to select tools, use tools, and generate code.

**Key Details:**

* The function uses a batch processing approach with a fixed batch size of 8.
* It calls the `agent.run` method for each task in batches, passing the task as input.
* It extracts the agent's answer from the response using the `extract_action` method.
* It evaluates the agent's code answer and compares it to the theoretical answer using the `evaluate_code` function.
* The scores are calculated and normalized to get the final tool selection score, tool used score, and code score.

**Dependencies and Interfaces:**

* `agent`: The evaluation agent with a toolbox containing tools such as `extract_action`, `run`.
* `EVALUATION_TASKS`: A list of tasks to evaluate.
* `TEST_TOOLS`: A set of expected tools in the test environment.

**Method Signatures:**

* `evaluate_agent(agent, batch_size=8, verbose=False, return_errors=False)`

Notable Variables:

* `tool_selection_score`
* `tool_used_score`
* `code_score`


key: chroma:document
string_value:
CODE:

def custom_print(*args):


SUMMARY:
**Summary:**

The `custom_print` function is a placeholder intended to print output, but it currently does nothing. Its primary purpose is overridden by the `return None` statement.

**Key Implementation Details:**

* Function signature: `def custom_print(*args):`
	+ Takes variable number of arguments (`*args`)
* Method signature:
  ```
  def __str__(object) -> str
```

**Notable Dependencies or Requirements:**

None mentioned, but it's likely that the function is intended to be part of a class with a `__str__` method.


key: chroma:document
string_value:
CODE:

def get_remote_tools(logger, organization="huggingface-tools"):
    if is_offline_mode():
        logger.info("You are in offline mode, so remote tools are not available.")
        return {}

    spaces = list_spaces(author=organization)
    tools = {}
    for space_info in spaces:
        repo_id = space_info.id
        resolved_config_file = hf_hub_download(repo_id, TOOL_CONFIG_FILE, repo_type="space")
        with open(resolved_config_file, encoding="utf-8") as reader:
            config = json.load(reader)
        task = repo_id.split("/")[-1]
        tools[config["name"]] = PreTool(
            task=task,
            description=config["description"],
            repo_id=repo_id,
            name=task,
            inputs=config["inputs"],
            output_type=config["output_type"],
        )



SUMMARY:
**Concise Summary**

The `get_remote_tools` function retrieves and initializes remote tasks for a given organization. It:

* Checks if the system is in offline mode, returning an empty dictionary if so.
* Downloads configuration files from the Hugging Face Hub using `hf_hub_download`.
* Parses JSON configurations to extract tool descriptions and input/output types.
* Creates instances of `PreTool` with task-specific data.

**Key Details**

* Uses `is_offline_mode()` function to check system status
* Downloads configuration files from Hugging Face Hub (`hf_hub_download`)
* Processes JSON configurations (using `json.load`)
* Initializes `PreTool` instances for each space repo

**Notable Dependencies/Requirements**

* `logger` object
* `spaces` list of space information
* `hf_hub_download` function to download configuration files from Hugging Face Hub
* `json` module for parsing JSON configurations
* `PreTool` class with initialized attributes


key: chroma:document
string_value:
CODE:

def setup_default_tools(logger):
    default_tools = {}
    main_module = importlib.import_module("transformers")
    tools_module = main_module.agents

    for task_name, tool_class_name in TOOL_MAPPING.items():
        tool_class = getattr(tools_module, tool_class_name)
        tool_instance = tool_class()
        default_tools[tool_class.name] = PreTool(
            name=tool_instance.name,
            inputs=tool_instance.inputs,
            output_type=tool_instance.output_type,
            task=task_name,
            description=tool_instance.description,
            repo_id=None,
        )



SUMMARY:
**Summary**

Establishes the default toolkit configuration by importing modules from `transformers` and populating a dictionary (`default_tools`) with instances of pre-defined tool classes. The tool class name is resolved using the `TOOL_MAPPING` and used to create an instance, which is then stored in the dictionary.

**Key Implementation Details**

* Iterates over `TOOL_MAPPING` dictionary
   * Resolves tool class name using `getattr(tools_module, tool_class_name)`
   * creates a new instance of the resolved class

**Notable Dependencies or Requirements**

* `transformers`: imported module from which top-level agents (tools) are retrieved.
* `ImportlibImportModule`: utilized to dynamically import modules.

**Function Signatures and Interfaces**

* `setup_default_tools(logger)`: function that takes a logger as an input parameter and returns the populated dictionary of default tools.


key: chroma:document
string_value:
CODE:

class PythonInterpreterTool(Tool):
    name = "python_interpreter"
    description = "This is a tool that evaluates python code. It can be used to perform calculations."

    output_type = "string"

    def __init__(self, *args, authorized_imports=None, **kwargs):
        if authorized_imports is None:
            self.authorized_imports = list(set(LIST_SAFE_MODULES))
        else:
            self.authorized_imports = list(set(LIST_SAFE_MODULES) | set(authorized_imports))
        self.inputs = {
            "code": {
                "type": "string",
                "description": (
                    "The code snippet to evaluate. All variables used in this snippet must be defined in this same snippet, "
                    f"else you will get an error. This code can only import the following python libraries: {authorized_imports}."
                ),
            }
        }
        super().__init__(*args, **kwargs)

    def forward(self, code):
        output = str(
            evaluate_python_code(code, static_tools=BASE_PYTHON_TOOLS, authorized_imports=self.authorized_imports)
        )


SUMMARY:
**PythonInterpreterTool Class Summary**

### Main Purpose and Functionality
The `PythonInterpreterTool` class evaluates Python code and performs calculations. It takes in a code snippet, checks for authorized imports, and returns the calculated result as a string.

### Key Implementation Details

* Uses `evaluate_python_code` function to execute code with checked imports.
* Settable output type and input field 'code' with description of its usage and allowed imports.

### Notable Dependencies or Requirements

* `BASE_PYTHON_TOOLS`: a set of predefined Python tools used for execution.
* `LIST_SAFE_MODULES`: a list of safe modules that can be imported without restriction.


key: chroma:document
string_value:
CODE:

class FinalAnswerTool(Tool):
    name = "final_answer"
    description = "Provides a final answer to the given problem."
    inputs = {"answer": {"type": "any", "description": "The final answer to the problem"}}
    output_type = "any"

    def forward(self, answer):


SUMMARY:
**Class Summary: FinalAnswerTool**

### Purpose and Functionality
A tool that takes a single input, `answer`, of type `any` and returns it as the final output.

### Key Implementation Details

* `forward(self, answer)` method, which simply passes the `answer` input to the output.
	+ Method signature: `forward(answer: any) -> any`
	+ Interface implemented: `Tool`

### Notable Dependencies or Requirements
None noted in the provided code snippet. However, it inherits from a base class `Tool`, suggesting that it depends on other tools being present in the overall system.

### Summary Summary:
The FinalAnswerTool is a simple tool that takes an "answer" input and returns it as output, with no significant algorithms or design choices applied.


key: chroma:document
string_value:
CODE:

class AgentType:
    """
    Abstract class to be reimplemented to define types that can be returned by agents.

    These objects serve three purposes:

    - They behave as they were the type they're meant to be, e.g., a string for text, a PIL.Image for images
    - They can be stringified: str(object) in order to return a string defining the object
    - They should be displayed correctly in ipython notebooks/colab/jupyter
    """

    def __init__(self, value):
        self._value = value

    def __str__(self):
        return self.to_string()

    def to_raw(self):
        logger.error(
            "This is a raw AgentType of unknown type. Display in notebooks and string conversion will be unreliable"
        )
        return self._value

    def to_string(self) -> str:
        logger.error(
            "This is a raw AgentType of unknown type. Display in notebooks and string conversion will be unreliable"
        )


SUMMARY:
**AgentType Class Summary**

The `AgentType` class is an abstract base class for defining types that can be returned by agents.

### Purpose
- Behaves like the type it represents (e.g., string for text, PIL.Image for images)
- Can be stringified to return a string defining the object
- Displays correctly in Jupyter notebooks/colab

### Key Implementation Details

* `__init__`: Initializes an AgentType with a value.
* `to_raw` and `to_string`: Return the raw value or convert it to a string, respectively, logging warnings for unknown types.

### Dependencies and Requirements
- Logger object for logging errors
- PIL.Image class (imported implicitly by using it)


key: chroma:document
string_value:
CODE:

class AgentText(AgentType, str):
    """
    Text type returned by the agent. Behaves as a string.
    """

    def to_raw(self):
        return self._value

    def to_string(self):


SUMMARY:
**Class Summary:**

`AgentText` is a customized `str` type representing text data, inheriting from `AgentType`.

**Primary Purpose:**
Wraps string value with additional functionality.

**Implementation Details:**

* Enforces polymorphic behavior through `to_raw` and `to_string` methods.
* `to_string` method explicitly converts to regular string type for consistency.

**Key Interfaces/Methods:**

* `to_raw(self):` Returns the underlying string value.
* `to_string(self):` Converts the value to a regular string instance.

**Notable Dependencies:**
Inherits from `AgentType`, which is not shown in this snippet.


key: chroma:document
string_value:
CODE:

class AgentImage(AgentType, ImageType):
    """
    Image type returned by the agent. Behaves as a PIL.Image.
    """

    def __init__(self, value):
        AgentType.__init__(self, value)
        ImageType.__init__(self)

        if not is_vision_available():
            raise ImportError("PIL must be installed in order to handle images.")

        self._path = None
        self._raw = None
        self._tensor = None

        if isinstance(value, ImageType):
            self._raw = value
        elif isinstance(value, (str, pathlib.Path)):
            self._path = value
        elif isinstance(value, torch.Tensor):
            self._tensor = value
        elif isinstance(value, np.ndarray):
            self._tensor = torch.from_numpy(value)
        else:
            raise TypeError(f"Unsupported type for {self.__class__.__name__}: {type(value)}")

    def _ipython_display_(self, include=None, exclude=None):
        """
        Displays correctly this type in an ipython notebook (ipython, colab, jupyter, ...)
        """
        from IPython.display import Image, display

        display(Image(self.to_string()))

    def to_raw(self):
        """
        Returns the "raw" version of that object. In the case of an AgentImage, it is a PIL.Image.
        """
        if self._raw is not None:
            return self._raw

        if self._path is not None:
            self._raw = Image.open(self._path)
            return self._raw

        if self._tensor is not None:
            array = self._tensor.cpu().detach().numpy()
            return Image.fromarray((255 - array * 255).astype(np.uint8))

    def to_string(self):
        """
        Returns the stringified version of that object. In the case of an AgentImage, it is a path to the serialized
        version of the image.
        """
        if self._path is not None:
            return self._path

        if self._raw is not None:
            directory = tempfile.mkdtemp()
            self._path = os.path.join(directory, str(uuid.uuid4()) + ".png")
            self._raw.save(self._path)
            return self._path

        if self._tensor is not None:
            array = self._tensor.cpu().detach().numpy()

            # There is likely simpler than load into image into save
            img = Image.fromarray((255 - array * 255).astype(np.uint8))

            directory = tempfile.mkdtemp()
            self._path = os.path.join(directory, str(uuid.uuid4()) + ".png")

            img.save(self._path)

            return self._path

    def save(self, output_bytes, format, **params):
        """
        Saves the image to a file.
        Args:
            output_bytes (bytes): The output bytes to save the image to.
            format (str): The format to use for the output image. The format is the same as in PIL.Image.save.
            **params: Additional parameters to pass to PIL.Image.save.
        """
        img = self.to_raw()


SUMMARY:
**Class Summary**

`AgentImage`: Combines `AgentType` and `ImageType` inheritance, treating images as a specific type of agent output that inherits PIL image behavior.

**Primary Functionality**

Represents images handled by an agent, providing methods for to_string(), _ipython_display(), and saving images in various formats.

**Key Features:**

1. **Initialization**: Initializes with `value`, handling different data types and raising errors for incorrect inputs.
2. **_to_raw()** method converts the image to PIL Image format if necessary.
3. **save()** method saves the image in a specified output format, parameterized via additional parameters.

**Important Attributes:**

* `path`: file path of the saved image
* `_raw`, `_tensor`: stored Raw Pil image and tensor versions of the input value respectively

**Notable Dependencies/Requirements:**

* PIL (image processing library)
* Torch (deep learning framework)

**Inherited Interfaces:**

* inherits from `AgentType`
* implements custom methods: _ipython_display_, to_string


key: chroma:document
string_value:
CODE:

class AgentAudio(AgentType, str):
    """
    Audio type returned by the agent.
    """

    def __init__(self, value, samplerate=16_000):
        super().__init__(value)

        if not is_soundfile_available():
            raise ImportError("soundfile must be installed in order to handle audio.")

        self._path = None
        self._tensor = None

        self.samplerate = samplerate
        if isinstance(value, (str, pathlib.Path)):
            self._path = value
        elif is_torch_available() and isinstance(value, torch.Tensor):
            self._tensor = value
        elif isinstance(value, tuple):
            self.samplerate = value[0]
            if isinstance(value[1], np.ndarray):
                self._tensor = torch.from_numpy(value[1])
            else:
                self._tensor = torch.tensor(value[1])
        else:
            raise ValueError(f"Unsupported audio type: {type(value)}")

    def _ipython_display_(self, include=None, exclude=None):
        """
        Displays correctly this type in an ipython notebook (ipython, colab, jupyter, ...)
        """
        from IPython.display import Audio, display

        display(Audio(self.to_string(), rate=self.samplerate))

    def to_raw(self):
        """
        Returns the "raw" version of that object. It is a `torch.Tensor` object.
        """
        if self._tensor is not None:
            return self._tensor

        if self._path is not None:
            tensor, self.samplerate = sf.read(self._path)
            self._tensor = torch.tensor(tensor)
            return self._tensor

    def to_string(self):
        """
        Returns the stringified version of that object. In the case of an AgentAudio, it is a path to the serialized
        version of the audio.
        """
        if self._path is not None:
            return self._path

        if self._tensor is not None:
            directory = tempfile.mkdtemp()
            self._path = os.path.join(directory, str(uuid.uuid4()) + ".wav")
            sf.write(self._path, self._tensor, samplerate=self.samplerate)


SUMMARY:
**Class Summary: AgentAudio**

**Purpose and Functionality**

`AgentAudio` is an audio type returned by the agent, implementing both `AgentType` and string interfaces. It represents an audio file or tensor.

**Key Implementation Details**

* Initializes with `value` (audio data) and optional `samplerate`.
* Checks for required dependencies: `soundfile` library.
* Supports various input formats:
	+ String or Path (audio path)
	+ Torch Tensor
	+ Tuple (sample rate, audio data)

**Notable Dependencies or Requirements**

* `torch`
* `soundfile`
* `pathlib`
* `IPython.display AUDIO`

**Interface Inheritance/ Implementation**

Extends `AgentType` and str interfaces.

**Key Method Signatures**

* `__init__(value, samplerate=16_000)`
* `_ipython_display_(include=None, exclude=None)`
* `to_raw() -> torch.Tensor`
* `to_string()`


key: chroma:document
string_value:
CODE:

def handle_agent_inputs(*args, **kwargs):
    args = [(arg.to_raw() if isinstance(arg, AgentType) else arg) for arg in args]
    kwargs = {k: (v.to_raw() if isinstance(v, AgentType) else v) for k, v in kwargs.items()}


SUMMARY:
**Summary:**

The `handle_agent_inputs` function processes input arguments and keyword arguments of a given function. It:

*   Normalizes input data by converting `AgentType` instances to their raw representation for consistent handling.
*   Returns the normalized argument list and dictionary. 

**Key implementation details:**

*   Uses list comprehensions to normalize individual arguments.
*   Applies normalization to both positional (`args`) and keyword (`kwargs`) inputs.

**Notable dependencies or requirements:**

*   Utilizes an `AgentType` class, implied by its appearance in the context.
*   Assumes a `to_raw()` method exists on `AgentType` instances.


key: chroma:document
string_value:
CODE:

def handle_agent_outputs(output, output_type=None):
    if output_type in AGENT_TYPE_MAPPING:
        # If the class has defined outputs, we can map directly according to the class definition
        decoded_outputs = AGENT_TYPE_MAPPING[output_type](output)
        return decoded_outputs
    else:
        # If the class does not have defined output, then we map according to the type
        for _k, _v in INSTANCE_TYPE_MAPPING.items():
            if isinstance(output, _k):
                return _v(output)


SUMMARY:
**Function Summary**

`handle_agent_outputs`: Decodes agent outputs based on associated class definitions or type mappings.

**Key Functionality:**

1. **Type-based Mapping**: Returns decoded outputs according to the specified `output_type`.
2. **Class-Defined Outputs**: Uses `AGENT_TYPE_MAPPING` dictionary to map outputs using classes defined in each `agent_type`.

**Implementation Details:**

* Utilizes `AGENT_TYPE_MAPPING` and `INSTANCE_TYPE_MAPPING` dictionaries.
* Supports both class-defined outputs and type-based mappings.

**Notable Dependencies:**

* `AGENT_type-MAPPING`: Dictionary mapping agent types to their respective output classes or functions.
* `INSTANCE_TYPE_MAPPING`: Dictionary mapping instance types to their corresponding conversion functions.


key: chroma:document
string_value:
CODE:

def get_repo_type(repo_id, repo_type=None, **hub_kwargs):
    if repo_type is not None:
        return repo_type
    try:
        hf_hub_download(repo_id, TOOL_CONFIG_FILE, repo_type="space", **hub_kwargs)
        return "space"
    except RepositoryNotFoundError:
        try:
            hf_hub_download(repo_id, TOOL_CONFIG_FILE, repo_type="model", **hub_kwargs)
            return "model"
        except RepositoryNotFoundError:
            raise EnvironmentError(f"`{repo_id}` does not seem to be a valid repo identifier on the Hub.")
        except Exception:
            return "model"
    except Exception:


SUMMARY:
**Summary**

`get_repo_type` is a function that determines the repository type for a given `repo_id`. It attempts to download the repository from Hub using two different types ("space" and "model") and returns the type of the first successful download. If both attempts fail, it raises an `EnvironmentError`.

**Key Implementation Details**

* Takes three parameters: `repo_id`, `repo_type` (optional), and `hub_kwargs` (keyword arguments).
* Uses `try-except` blocks to handle different exceptions:
	+ `RepositoryNotFoundError`: attempts ` repo_type="space"` or "model" until success.
	+ General exceptions are caught and default to "model".
* Returns the type of the repository as a string ("space", "model", or an error message).

**Notable Dependencies/Requirements**

* `hf_hub_download` function ( referenced without implementation)
* `TOOL_CONFIG_FILE` variable (referenced but not defined in this snippet)


key: chroma:document
string_value:
CODE:

def validate_after_init(cls, do_validate_forward: bool = True):
    original_init = cls.__init__

    @wraps(original_init)
    def new_init(self, *args, **kwargs):
        original_init(self, *args, **kwargs)
        if not isinstance(self, PipelineTool):
            self.validate_arguments(do_validate_forward=do_validate_forward)

    cls.__init__ = new_init


SUMMARY:
**Summary**

A class-level wrapper function `validate_after_init` that modifies the `__init__` method to validate instance arguments after initialization. It wraps the original `__init__` method using `@wraps` and introduces additional validation if `do_validate_forward` is true.

**Key Points:**

* Original `__init__` method is preserved and wrapped.
* Validation of instance arguments occurs after initialization.
* `validate_arguments` method is called with optional `do_validate_forward` parameter.
* Replacement `new_init` method modifies the class's `__init__` attribute.

**Note:** No external dependencies listed.


key: chroma:document
string_value:
CODE:

class Tool:
    """
    A base class for the functions used by the agent. Subclass this and implement the `__call__` method as well as the
    following class attributes:

    - **description** (`str`) -- A short description of what your tool does, the inputs it expects and the output(s) it
      will return. For instance 'This is a tool that downloads a file from a `url`. It takes the `url` as input, and
      returns the text contained in the file'.
    - **name** (`str`) -- A performative name that will be used for your tool in the prompt to the agent. For instance
      `"text-classifier"` or `"image_generator"`.
    - **inputs** (`Dict[str, Dict[str, Union[str, type]]]`) -- The dict of modalities expected for the inputs.
      It has one `type`key and a `description`key.
      This is used by `launch_gradio_demo` or to make a nice space from your tool, and also can be used in the generated
      description for your tool.
    - **output_type** (`type`) -- The type of the tool output. This is used by `launch_gradio_demo`
      or to make a nice space from your tool, and also can be used in the generated description for your tool.

    You can also override the method [`~Tool.setup`] if your tool as an expensive operation to perform before being
    usable (such as loading a model). [`~Tool.setup`] will be called the first time you use your tool, but not at
    instantiation.
    """

    name: str
    description: str
    inputs: Dict[str, Dict[str, Union[str, type]]]
    output_type: type

    def __init__(self, *args, **kwargs):
        self.is_initialized = False

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        validate_after_init(cls, do_validate_forward=False)

    def validate_arguments(self, do_validate_forward: bool = True):
        required_attributes = {
            "description": str,
            "name": str,
            "inputs": dict,
            "output_type": str,
        }
        authorized_types = ["string", "integer", "number", "image", "audio", "any", "boolean"]

        for attr, expected_type in required_attributes.items():
            attr_value = getattr(self, attr, None)
            if attr_value is None:
                raise TypeError(f"You must set an attribute {attr}.")
            if not isinstance(attr_value, expected_type):
                raise TypeError(
                    f"Attribute {attr} should have type {expected_type.__name__}, got {type(attr_value)} instead."
                )
        for input_name, input_content in self.inputs.items():
            assert isinstance(input_content, dict), f"Input '{input_name}' should be a dictionary."
            assert (
                "type" in input_content and "description" in input_content
            ), f"Input '{input_name}' should have keys 'type' and 'description', has only {list(input_content.keys())}."
            if input_content["type"] not in authorized_types:
                raise Exception(
                    f"Input '{input_name}': type '{input_content['type']}' is not an authorized value, should be one of {authorized_types}."
                )

        assert getattr(self, "output_type", None) in authorized_types
        if do_validate_forward:
            if not isinstance(self, PipelineTool):
                signature = inspect.signature(self.forward)
                if not set(signature.parameters.keys()) == set(self.inputs.keys()):
                    raise Exception(
                        "Tool's 'forward' method should take 'self' as its first argument, then its next arguments should match the keys of tool attribute 'inputs'."
                    )

    def forward(self, *args, **kwargs):
        return NotImplemented("Write this method in your subclass of `Tool`.")

    def __call__(self, *args, **kwargs):
        args, kwargs = handle_agent_inputs(*args, **kwargs)
        outputs = self.forward(*args, **kwargs)
        return handle_agent_outputs(outputs, self.output_type)

    def setup(self):
        """
        Overwrite this method here for any operation that is expensive and needs to be executed before you start using
        your tool. Such as loading a big model.
        """
        self.is_initialized = True

    def save(self, output_dir):
        """
        Saves the relevant code files for your tool so it can be pushed to the Hub. This will copy the code of your
        tool in `output_dir` as well as autogenerate:

        - a config file named `tool_config.json`
        - an `app.py` file so that your tool can be converted to a space
        - a `requirements.txt` containing the names of the module used by your tool (as detected when inspecting its
          code)

        You should only use this method to save tools that are defined in a separate module (not `__main__`).

        Args:
            output_dir (`str`): The folder in which you want to save your tool.
        """
        os.makedirs(output_dir, exist_ok=True)
        # Save module file
        if self.__module__ == "__main__":
            raise ValueError(
                f"We can't save the code defining {self} in {output_dir} as it's been defined in __main__. You "
                "have to put this code in a separate module so we can include it in the saved folder."
            )
        module_files = custom_object_save(self, output_dir)

        module_name = self.__class__.__module__
        last_module = module_name.split(".")[-1]
        full_name = f"{last_module}.{self.__class__.__name__}"

        # Save config file
        config_file = os.path.join(output_dir, "tool_config.json")
        if os.path.isfile(config_file):
            with open(config_file, "r", encoding="utf-8") as f:
                tool_config = json.load(f)
        else:
            tool_config = {}

        tool_config = {
            "tool_class": full_name,
            "description": self.description,
            "name": self.name,
            "inputs": self.inputs,
            "output_type": str(self.output_type),
        }
        with open(config_file, "w", encoding="utf-8") as f:
            f.write(json.dumps(tool_config, indent=2, sort_keys=True) + "\n")

        # Save app file
        app_file = os.path.join(output_dir, "app.py")
        with open(app_file, "w", encoding="utf-8") as f:
            f.write(APP_FILE_TEMPLATE.format(module_name=last_module, class_name=self.__class__.__name__))

        # Save requirements file
        requirements_file = os.path.join(output_dir, "requirements.txt")
        imports = []
        for module in module_files:
            imports.extend(get_imports(module))
        imports = list(set(imports))
        with open(requirements_file, "w", encoding="utf-8") as f:
            f.write("\n".join(imports) + "\n")

    @classmethod
    def from_hub(
        cls,
        repo_id: str,
        token: Optional[str] = None,
        **kwargs,
    ):
        """
        Loads a tool defined on the Hub.

        <Tip warning={true}>

        Loading a tool from the Hub means that you'll download the tool and execute it locally.
        ALWAYS inspect the tool you're downloading before loading it within your runtime, as you would do when
        installing a package using pip/npm/apt.

        </Tip>

        Args:
            repo_id (`str`):
                The name of the repo on the Hub where your tool is defined.
            token (`str`, *optional*):
                The token to identify you on hf.co. If unset, will use the token generated when running
                `huggingface-cli login` (stored in `~/.huggingface`).
            kwargs (additional keyword arguments, *optional*):
                Additional keyword arguments that will be split in two: all arguments relevant to the Hub (such as
                `cache_dir`, `revision`, `subfolder`) will be used when downloading the files for your tool, and the
                others will be passed along to its init.
        """
        hub_kwargs_names = [
            "cache_dir",
            "force_download",
            "resume_download",
            "proxies",
            "revision",
            "repo_type",
            "subfolder",
            "local_files_only",
        ]
        hub_kwargs = {k: v for k, v in kwargs.items() if k in hub_kwargs_names}

        # Try to get the tool config first.
        hub_kwargs["repo_type"] = get_repo_type(repo_id, **hub_kwargs)
        resolved_config_file = cached_file(
            repo_id,
            TOOL_CONFIG_FILE,
            token=token,
            **hub_kwargs,
            _raise_exceptions_for_gated_repo=False,
            _raise_exceptions_for_missing_entries=False,
            _raise_exceptions_for_connection_errors=False,
        )
        is_tool_config = resolved_config_file is not None
        if resolved_config_file is None:
            resolved_config_file = cached_file(
                repo_id,
                CONFIG_NAME,
                token=token,
                **hub_kwargs,
                _raise_exceptions_for_gated_repo=False,
                _raise_exceptions_for_missing_entries=False,
                _raise_exceptions_for_connection_errors=False,
            )
        if resolved_config_file is None:
            raise EnvironmentError(
                f"{repo_id} does not appear to provide a valid configuration in `tool_config.json` or `config.json`."
            )

        with open(resolved_config_file, encoding="utf-8") as reader:
            config = json.load(reader)

        if not is_tool_config:
            if "custom_tool" not in config:
                raise EnvironmentError(
                    f"{repo_id} does not provide a mapping to custom tools in its configuration `config.json`."
                )
            custom_tool = config["custom_tool"]
        else:
            custom_tool = config

        tool_class = custom_tool["tool_class"]
        tool_class = get_class_from_dynamic_module(tool_class, repo_id, token=token, **hub_kwargs)

        if len(tool_class.name) == 0:
            tool_class.name = custom_tool["name"]
        if tool_class.name != custom_tool["name"]:
            logger.warning(
                f"{tool_class.__name__} implements a different name in its configuration and class. Using the tool "
                "configuration name."
            )
            tool_class.name = custom_tool["name"]

        if len(tool_class.description) == 0:
            tool_class.description = custom_tool["description"]
        if tool_class.description != custom_tool["description"]:
            logger.warning(
                f"{tool_class.__name__} implements a different description in its configuration and class. Using the "
                "tool configuration description."
            )
            tool_class.description = custom_tool["description"]

        if tool_class.inputs != custom_tool["inputs"]:
            tool_class.inputs = custom_tool["inputs"]
        if tool_class.output_type != custom_tool["output_type"]:
            tool_class.output_type = custom_tool["output_type"]

        if not isinstance(tool_class.inputs, dict):
            tool_class.inputs = ast.literal_eval(tool_class.inputs)

        return tool_class(**kwargs)

    def push_to_hub(
        self,
        repo_id: str,
        commit_message: str = "Upload tool",
        private: Optional[bool] = None,
        token: Optional[Union[bool, str]] = None,
        create_pr: bool = False,
    ) -> str:
        """
        Upload the tool to the Hub.

        For this method to work properly, your tool must have been defined in a separate module (not `__main__`).
        For instance:
        ```
        from my_tool_module import MyTool
        my_tool = MyTool()
        my_tool.push_to_hub("my-username/my-space")
        ```

        Parameters:
            repo_id (`str`):
                The name of the repository you want to push your tool to. It should contain your organization name when
                pushing to a given organization.
            commit_message (`str`, *optional*, defaults to `"Upload tool"`):
                Message to commit while pushing.
            private (`bool`, *optional*):
                Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists.
            token (`bool` or `str`, *optional*):
                The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated
                when running `huggingface-cli login` (stored in `~/.huggingface`).
            create_pr (`bool`, *optional*, defaults to `False`):
                Whether or not to create a PR with the uploaded files or directly commit.
        """
        repo_url = create_repo(
            repo_id=repo_id,
            token=token,
            private=private,
            exist_ok=True,
            repo_type="space",
            space_sdk="gradio",
        )
        repo_id = repo_url.repo_id
        metadata_update(repo_id, {"tags": ["tool"]}, repo_type="space")

        with tempfile.TemporaryDirectory() as work_dir:
            # Save all files.
            self.save(work_dir)
            logger.info(f"Uploading the following files to {repo_id}: {','.join(os.listdir(work_dir))}")
            return upload_folder(
                repo_id=repo_id,
                commit_message=commit_message,
                folder_path=work_dir,
                token=token,
                create_pr=create_pr,
                repo_type="space",
            )

    @staticmethod
    def from_space(
        space_id: str, name: str, description: str, api_name: Optional[str] = None, token: Optional[str] = None
    ):
        """
        Creates a [`Tool`] from a Space given its id on the Hub.

        Args:
            space_id (`str`):
                The id of the Space on the Hub.
            name (`str`):
                The name of the tool.
            description (`str`):
                The description of the tool.
            api_name (`str`, *optional*):
                The specific api_name to use, if the space has several tabs. If not precised, will default to the first available api.
            token (`str`, *optional*):
                Add your token to access private spaces or increase your GPU quotas.
        Returns:
            [`Tool`]:
                The Space, as a tool.

        Examples:
        ```
        image_generator = Tool.from_space(
            space_id="black-forest-labs/FLUX.1-schnell",
            name="image-generator",
            description="Generate an image from a prompt"
        )
        image = image_generator("Generate an image of a cool surfer in Tahiti")
        ```
        ```
        face_swapper = Tool.from_space(
            "tuan2308/face-swap",
            "face_swapper",
            "Tool that puts the face shown on the first image on the second image. You can give it paths to images.",
        )
        image = face_swapper('./aymeric.jpeg', './ruth.jpg')
        ```
        """
        from gradio_client import Client, handle_file
        from gradio_client.utils import is_http_url_like

        class SpaceToolWrapper(Tool):
            def __init__(
                self,
                space_id: str,
                name: str,
                description: str,
                api_name: Optional[str] = None,
                token: Optional[str] = None,
            ):
                self.client = Client(space_id, hf_token=token)
                self.name = name
                self.description = description
                space_description = self.client.view_api(return_format="dict", print_info=False)["named_endpoints"]

                # If api_name is not defined, take the first of the available APIs for this space
                if api_name is None:
                    api_name = list(space_description.keys())[0]
                    logger.warning(
                        f"Since `api_name` was not defined, it was automatically set to the first avilable API: `{api_name}`."
                    )
                self.api_name = api_name

                try:
                    space_description_api = space_description[api_name]
                except KeyError:
                    raise KeyError(f"Could not find specified {api_name=} among available api names.")

                self.inputs = {}
                for parameter in space_description_api["parameters"]:
                    if not parameter["parameter_has_default"]:
                        parameter_type = parameter["type"]["type"]
                        if parameter_type == "object":
                            parameter_type = "any"
                        self.inputs[parameter["parameter_name"]] = {
                            "type": parameter_type,
                            "description": parameter["python_type"]["description"],
                        }
                output_component = space_description_api["returns"][0]["component"]
                if output_component == "Image":
                    self.output_type = "image"
                elif output_component == "Audio":
                    self.output_type = "audio"
                else:
                    self.output_type = "any"

            def sanitize_argument_for_prediction(self, arg):
                if isinstance(arg, ImageType):
                    temp_file = tempfile.NamedTemporaryFile(suffix=".png", delete=False)
                    arg.save(temp_file.name)
                    arg = temp_file.name
                if (isinstance(arg, (str, Path)) and Path(arg).exists() and Path(arg).is_file()) or is_http_url_like(
                    arg
                ):
                    arg = handle_file(arg)
                return arg

            def forward(self, *args, **kwargs):
                # Preprocess args and kwargs:
                args = list(args)
                for i, arg in enumerate(args):
                    args[i] = self.sanitize_argument_for_prediction(arg)
                for arg_name, arg in kwargs.items():
                    kwargs[arg_name] = self.sanitize_argument_for_prediction(arg)

                output = self.client.predict(*args, api_name=self.api_name, **kwargs)
                if isinstance(output, tuple) or isinstance(output, list):
                    return output[
                        0
                    ]  # Sometime the space also returns the generation seed, in which case the result is at index 0
                return output

        return SpaceToolWrapper(space_id, name, description, api_name=api_name, token=token)

    @staticmethod
    def from_gradio(gradio_tool):
        """
        Creates a [`Tool`] from a gradio tool.
        """
        import inspect

        class GradioToolWrapper(Tool):
            def __init__(self, _gradio_tool):
                self.name = _gradio_tool.name
                self.description = _gradio_tool.description
                self.output_type = "string"
                self._gradio_tool = _gradio_tool
                func_args = list(inspect.signature(_gradio_tool.run).parameters.items())
                self.inputs = {
                    key: {"type": CONVERSION_DICT[value.annotation], "description": ""} for key, value in func_args
                }
                self.forward = self._gradio_tool.run

        return GradioToolWrapper(gradio_tool)

    @staticmethod
    def from_langchain(langchain_tool):
        """
        Creates a [`Tool`] from a langchain tool.
        """

        class LangChainToolWrapper(Tool):
            def __init__(self, _langchain_tool):
                self.name = _langchain_tool.name.lower()
                self.description = _langchain_tool.description
                self.inputs = _langchain_tool.args.copy()
                for input_content in self.inputs.values():
                    if "title" in input_content:
                        input_content.pop("title")
                    input_content["description"] = ""
                self.output_type = "string"
                self.langchain_tool = _langchain_tool

            def forward(self, *args, **kwargs):
                tool_input = kwargs.copy()
                for index, argument in enumerate(args):
                    if index < len(self.inputs):
                        input_key = next(iter(self.inputs))
                        tool_input[input_key] = argument
                return self.langchain_tool.run(tool_input)



SUMMARY:
This is the implementation of the `Tool` class in Python, which is used to represent and interact with different types of AI models. The class provides a common interface for these models, making it easier to integrate them into other systems.

The class has several key features:

*   It inherits from a base class (not shown in this code snippet) that likely provides basic functionality for all tools.
*   Each tool has a unique name, description, output type, and input structure.
*   The `forward` method is used to make predictions or execute the model's logic. This method takes in arguments and passes them to the corresponding model.
*   Some classes (e.g., `SpaceToolWrapper`, `GradioToolWrapper`) wrap around a specific type of tool.

Overall, this code provides a flexible framework for building and using AI models.


key: chroma:document
string_value:
CODE:

def get_tool_description_with_args(tool: Tool, description_template: str = DEFAULT_TOOL_DESCRIPTION_TEMPLATE) -> str:
    compiled_template = compile_jinja_template(description_template)
    rendered = compiled_template.render(
        tool=tool,
    )


SUMMARY:
**Function Summary**

`get_tool_description_with_args`: Compiles and renders a Jinja template with the given `Tool` object to produce a formatted string description.

**Key Details**

* Compiles a Jinja template using `compile_jinja_template(description_template)`
* Renders the template with the provided `tool` object
* Renders parameters: `tool`
* Returns a rendered string description

**Notable Dependencies/Requirements**

* `Tool` class/instance (expected)
* External dependency on `compile_jinja_template` and `render` from Jinja templating engine


