# embedding_metadata

key: chroma:document
string_value:
CODE:

class AutoQuantizationConfig:
    """
    The Auto-HF quantization config class that takes care of automatically dispatching to the correct
    quantization config given a quantization config stored in a dictionary.
    """

    @classmethod
    def from_dict(cls, quantization_config_dict: Dict):
        quant_method = quantization_config_dict.get("quant_method", None)
        # We need a special care for bnb models to make sure everything is BC ..
        if quantization_config_dict.get("load_in_8bit", False) or quantization_config_dict.get("load_in_4bit", False):
            suffix = "_4bit" if quantization_config_dict.get("load_in_4bit", False) else "_8bit"
            quant_method = QuantizationMethod.BITS_AND_BYTES + suffix
        elif quant_method is None:
            raise ValueError(
                "The model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantized"
            )

        if quant_method not in AUTO_QUANTIZATION_CONFIG_MAPPING.keys():
            raise ValueError(
                f"Unknown quantization type, got {quant_method} - supported types are:"
                f" {list(AUTO_QUANTIZER_MAPPING.keys())}"
            )

        target_cls = AUTO_QUANTIZATION_CONFIG_MAPPING[quant_method]
        return target_cls.from_dict(quantization_config_dict)

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
        model_config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
        if getattr(model_config, "quantization_config", None) is None:
            raise ValueError(
                f"Did not found a `quantization_config` in {pretrained_model_name_or_path}. Make sure that the model is correctly quantized."
            )
        quantization_config_dict = model_config.quantization_config
        quantization_config = cls.from_dict(quantization_config_dict)
        # Update with potential kwargs that are passed through from_pretrained.
        quantization_config.update(**kwargs)


SUMMARY:
**AutoQuantizationConfig Class Summary**

The `AutoQuantizationConfig` class automates the creation of quantization config objects based on input dictionaries and pre-trained models. It dispatches to the correct quantization method using a predefined mapping.

**Main Purpose and Functionality:**
Dispatches to the correct quantization method based on the input dictionary or pre-trained model.

**Key Implementation Details:**

* `from_dict`: Accepts a quantization config dictionary, verifies the presence of `quant_method` and raises errors if necessary.
* `from_pretrained`: Extracts the quantization config from a pre-trained model and updates it with additional kwargs.

**Notable Dependencies and Requirements:**

* `AUTO_QUANTIZATION_CONFIG_MAPPING`
* `AUTO_QUANTIZER_MAPPING`
* `AutoConfig`

**Important Method Signatures and Interfaces:**

* `from_dict(quantization_config_dict: Dict)`
* `from_pretrained(pretrained_model_name_or_path, **kwargs)`


key: chroma:document
string_value:
CODE:

class AutoHfQuantizer:
    """
     The Auto-HF quantizer class that takes care of automatically instantiating to the correct
    `HfQuantizer` given the `QuantizationConfig`.
    """

    @classmethod
    def from_config(cls, quantization_config: Union[QuantizationConfigMixin, Dict], **kwargs):
        # Convert it to a QuantizationConfig if the q_config is a dict
        if isinstance(quantization_config, dict):
            quantization_config = AutoQuantizationConfig.from_dict(quantization_config)

        quant_method = quantization_config.quant_method

        # Again, we need a special care for bnb as we have a single quantization config
        # class for both 4-bit and 8-bit quantization
        if quant_method == QuantizationMethod.BITS_AND_BYTES:
            if quantization_config.load_in_8bit:
                quant_method += "_8bit"
            else:
                quant_method += "_4bit"

        if quant_method not in AUTO_QUANTIZER_MAPPING.keys():
            raise ValueError(
                f"Unknown quantization type, got {quant_method} - supported types are:"
                f" {list(AUTO_QUANTIZER_MAPPING.keys())}"
            )

        target_cls = AUTO_QUANTIZER_MAPPING[quant_method]
        return target_cls(quantization_config, **kwargs)

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
        quantization_config = AutoQuantizationConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
        return cls.from_config(quantization_config)

    @classmethod
    def merge_quantization_configs(
        cls,
        quantization_config: Union[dict, QuantizationConfigMixin],
        quantization_config_from_args: Optional[QuantizationConfigMixin],
    ):
        """
        handles situations where both quantization_config from args and quantization_config from model config are present.
        """
        if quantization_config_from_args is not None:
            warning_msg = (
                "You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading"
                " already has a `quantization_config` attribute. The `quantization_config` from the model will be used."
            )
        else:
            warning_msg = ""

        if isinstance(quantization_config, dict):
            quantization_config = AutoQuantizationConfig.from_dict(quantization_config)

        if (
            isinstance(quantization_config, (GPTQConfig, AwqConfig, FbgemmFp8Config, CompressedTensorsConfig))
            and quantization_config_from_args is not None
        ):
            # special case for GPTQ / AWQ / FbgemmFp8 config collision
            loading_attr_dict = quantization_config_from_args.get_loading_attributes()
            for attr, val in loading_attr_dict.items():
                setattr(quantization_config, attr, val)

            warning_msg += f"However, loading attributes (e.g. {list(loading_attr_dict.keys())}) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored."

        if warning_msg != "":
            warnings.warn(warning_msg)



SUMMARY:
**AutoHfQuantizer Class Summary**

### Purpose and Functionality
The AutoHfQuantizer class automatcially instantiates the correct `HfQuantizer` given a `QuantizationConfig`. It provides methods for creating instances from different sources, including configuration files and pre-trained models.

### Key Implementation Details

*   Converts configuration to `AutoQuantizationConfig` if necessary.
*   Handles special cases for bnb (both 4-bit and 8-bit) quantization.
*   Uses the `AUTO_QUANTIZER_MAPPING` dictionary to map quantization methods to their corresponding classes.
*   Provides a method for merging two quantization configurations.

### Notable Dependencies or Requirements

*   `AutoQuantizationConfig`
*   `HfQuantizer`
*   `AUTO_QUANTIZER_MAPPING` dictionary
*   `QuantizationMethod.BITS_AND_BYTES`

```markdown
# Class Name: AutoHfQuantizer

## Purpose and Functionality
Creates instances of HfQuantizer given a QuantizationConfig.

## Key Implementation Details
- Converts configuration to AutoQuantizationConfig.
- Handles special case for bnb quantization.
- Uses AUTO_QUANTIZER_MAPPING dictionary.

## Notable Dependencies or Requirements
- AutoQuantizationConfig
- HfQuantizer
- AUTO_QUANTIFIER_MAPPING dictionary
```


key: chroma:document
string_value:
CODE:

class HfQuantizer(ABC):
    """
    Abstract class of the HuggingFace quantizer. Supports for now quantizing HF transformers models for inference and/or quantization.
    This class is used only for transformers.PreTrainedModel.from_pretrained and cannot be easily used outside the scope of that method
    yet.

    Attributes
        quantization_config (`transformers.utils.quantization_config.QuantizationConfigMixin`):
            The quantization config that defines the quantization parameters of your model that you want to quantize.
        modules_to_not_convert (`List[str]`, *optional*):
            The list of module names to not convert when quantizing the model.
        required_packages (`List[str]`, *optional*):
            The list of required pip packages to install prior to using the quantizer
        requires_calibration (`bool`):
            Whether the quantization method requires to calibrate the model before using it.
        requires_parameters_quantization (`bool`):
            Whether the quantization method requires to create a new Parameter. For example, for bitsandbytes, it is
            required to create a new xxxParameter in order to properly quantize the model.
    """

    requires_calibration = False
    required_packages = None
    requires_parameters_quantization = False

    def __init__(self, quantization_config: QuantizationConfigMixin, **kwargs):
        self.quantization_config = quantization_config

        # -- Handle extra kwargs below --
        self.modules_to_not_convert = kwargs.pop("modules_to_not_convert", [])
        self.pre_quantized = kwargs.pop("pre_quantized", True)

        if not self.pre_quantized and self.requires_calibration:
            raise ValueError(
                f"The quantization method {quantization_config.quant_method} does require the model to be pre-quantized."
                f" You explicitly passed `pre_quantized=False` meaning your model weights are not quantized. Make sure to "
                f"pass `pre_quantized=True` while knowing what you are doing."
            )

    def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
        """
        Some quantization methods require to explicitly set the dtype of the model to a
        target dtype. You need to override this method in case you want to make sure that behavior is
        preserved

        Args:
            torch_dtype (`torch.dtype`):
                The input dtype that is passed in `from_pretrained`
        """
        return torch_dtype

    def update_device_map(self, device_map: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]:
        """
        Override this method if you want to pass a override the existing device map with a new
        one. E.g. for bitsandbytes, since `accelerate` is a hard requirement, if no device_map is
        passed, the device_map is set to `"auto"``

        Args:
            device_map (`Union[dict, str]`, *optional*):
                The device_map that is passed through the `from_pretrained` method.
        """
        return device_map

    def adjust_target_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
        """
        Override this method if you want to adjust the `target_dtype` variable used in `from_pretrained`
        to compute the device_map in case the device_map is a `str`. E.g. for bitsandbytes we force-set `target_dtype`
        to `torch.int8` and for 4-bit we pass a custom enum `accelerate.CustomDtype.int4`.

        Args:
            torch_dtype (`torch.dtype`, *optional*):
                The torch_dtype that is used to compute the device_map.
        """
        return torch_dtype

    def update_missing_keys(self, model, missing_keys: List[str], prefix: str) -> List[str]:
        """
        Override this method if you want to adjust the `missing_keys`.

        Args:
            missing_keys (`List[str]`, *optional*):
                The list of missing keys in the checkpoint compared to the state dict of the model
        """
        return missing_keys

    def update_expected_keys(self, model, expected_keys: List[str], loaded_keys: List[str]) -> List[str]:
        """
        Override this method if you want to adjust the `update_expected_keys`.

        Args:
            expected_keys (`List[str]`, *optional*):
                The list of the expected keys in the initialized model.
            loaded_keys (`List[str]`, *optional*):
                The list of the loaded keys in the checkpoint.
        """
        return expected_keys

    def get_special_dtypes_update(self, model, torch_dtype: "torch.dtype") -> Dict[str, "torch.dtype"]:
        """
        returns dtypes for modules that are not quantized - used for the computation of the device_map in case
        one passes a str as a device_map. The method will use the `modules_to_not_convert` that is modified
        in `_process_model_before_weight_loading`.

        Args:
            model (`~transformers.PreTrainedModel`):
                The model to quantize
            torch_dtype (`torch.dtype`):
                The dtype passed in `from_pretrained` method.
        """

        return {
            name: torch_dtype
            for name, _ in model.named_parameters()
            if any(m in name for m in self.modules_to_not_convert)
        }

    def adjust_max_memory(self, max_memory: Dict[str, Union[int, str]]) -> Dict[str, Union[int, str]]:
        """adjust max_memory argument for infer_auto_device_map() if extra memory is needed for quantization"""
        return max_memory

    def check_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        state_dict: Dict[str, Any],
        **kwargs,
    ) -> bool:
        """
        checks if a loaded state_dict component is part of quantized param + some validation; only defined if
        requires_parameters_quantization == True for quantization methods that require to create a new parameters
        for quantization.
        """
        return False

    def create_quantized_param(self, *args, **kwargs) -> "torch.nn.Parameter":
        """
        takes needed components from state_dict and creates quantized param; only applicable if
        requires_parameters_quantization == True
        """
        if not self.requires_parameters_quantization:
            raise AttributeError(
                f"`.create_quantized_param()` method is not supported by quantizer class {self.__class__.__name__}."
            )

    def validate_environment(self, *args, **kwargs):
        """
        This method is used to potentially check for potential conflicts with arguments that are
        passed in `from_pretrained`. You need to define it for all future quantizers that are integrated with transformers.
        If no explicit check are needed, simply return nothing.
        """
        return

    def preprocess_model(self, model: "PreTrainedModel", **kwargs):
        """
        Setting model attributes and/or converting model before weights loading. At this point
        the model should be initialized on the meta device so you can freely manipulate the skeleton
        of the model in order to replace modules in-place. Make sure to override the abstract method `_process_model_before_weight_loading`.

        Args:
            model (`~transformers.PreTrainedModel`):
                The model to quantize
            kwargs (`dict`, *optional*):
                The keyword arguments that are passed along `_process_model_before_weight_loading`.
        """
        model.is_quantized = True
        model.quantization_method = self.quantization_config.quant_method
        return self._process_model_before_weight_loading(model, **kwargs)

    def postprocess_model(self, model: "PreTrainedModel", **kwargs):
        """
        Post-process the model post weights loading.
        Make sure to override the abstract method `_process_model_after_weight_loading`.

        Args:
            model (`~transformers.PreTrainedModel`):
                The model to quantize
            kwargs (`dict`, *optional*):
                The keyword arguments that are passed along `_process_model_after_weight_loading`.
        """
        return self._process_model_after_weight_loading(model, **kwargs)

    def dequantize(self, model):
        """
        Potentially dequantize the model to retrive the original model, with some loss in accuracy / performance.
        Note not all quantization schemes support this.
        """
        model = self._dequantize(model)

        # Delete quantizer and quantization config
        del model.hf_quantizer
        del model.config.quantization_config
        del model.config._pre_quantization_dtype
        model.is_quantized = False

        return model

    def _dequantize(self, model):
        raise NotImplementedError(
            f"{self.quantization_config.quant_method} has no implementation of `dequantize`, please raise an issue on GitHub."
        )

    @property
    def is_qat_trainable(self) -> bool:
        """Flag indicating whether the quantized model can carry out quantization aware training"""
        return False

    @abstractmethod
    def _process_model_before_weight_loading(self, model, **kwargs): ...

    @abstractmethod
    def _process_model_after_weight_loading(self, model, **kwargs): ...

    @abstractmethod
    def is_serializable(self, safe_serialization=None): ...

    @property
    @abstractmethod


SUMMARY:
This is an abstract Python class `HfQuantizer` that provides the base functionality for quantizing Hugging Face transformer models. It's designed to be subclassed by specific quantization methods.

The class has several key features:

1.  **Quantization configuration**: The class takes a `quantization_config` object as input, which contains settings such as the quantization method (e.g., int8, float16) and any additional parameters required for the chosen method.
2.  **Model preprocessing**: The class provides methods to preprocess the model before weights loading (`preprocess_model`) and after weights loading (`postprocess_model`). These methods allow the user to manipulate the model's skeleton or override certain processing steps.
3.  **Dequantization**: The class has a `dequantize` method that can potentially dequantize the model to retrieve its original values, although this may result in reduced accuracy or performance.
4.  **Validation and serialization**: The class includes methods for checking whether the quantized model can perform gradient-based training (`check_quantized_param`) and validating the environment for potential conflicts with other arguments passed during `from_pretrained`.
5.  **Quantization aware training (QAT)**: While the class doesn't currently support QAT, it defines an abstract method `_is_qat_trainable` to be overridden in subclasses.

Overall, this class provides a flexible and modular framework for quantizing transformer models based on specific requirements for various types of quantization methods.


key: chroma:document
string_value:
CODE:

class BitNetHfQuantizer(HfQuantizer):
    """
    1.58-bit quantization from BitNet quantization method:
    Before loading: it converts the linear layers into BitLinear layers during loading.

    Checkout the paper introducing this method : https://arxiv.org/pdf/2402.17764
    """

    requires_parameters_quantization = False
    requires_calibration = True

    required_packages = ["accelerate"]

    def __init__(self, quantization_config, **kwargs):
        super().__init__(quantization_config, **kwargs)
        self.quantization_config = quantization_config

    def validate_environment(self, *args, **kwargs):
        if not is_accelerate_available():
            raise ImportError("Loading a BitNet quantized model requires accelerate (`pip install accelerate`)")

        if kwargs.get("from_tf", False) or kwargs.get("from_flax", False):
            raise ValueError(
                "Loading ternary weights from tf/flax is currently not supported, please make"
                " sure the weights are in PyTorch format."
            )

        if not torch.cuda.is_available():
            logger.warning_once(
                "You don't have a GPU available to load the model, the inference will be slow because of weight unpacking"
            )
            return

        device_map = kwargs.get("device_map", None)
        if device_map is None:
            logger.warning_once(
                "You have loaded a BitNet model on CPU and have a CUDA device available, make sure to set "
                "your model on a GPU device in order to run your model."
            )
        elif device_map is not None:
            if isinstance(device_map, dict) and ("cpu" in device_map.values() or "disk" in device_map.values()):
                raise ValueError(
                    "You are attempting to load a BitNet model with a device_map that contains a CPU or disk device."
                    "This is not supported. Please remove the CPU or disk device from the device_map."
                )

    def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
        return model

    def _process_model_before_weight_loading(
        self,
        model: "PreTrainedModel",
        device_map,
        keep_in_fp32_modules: List[str] = [],
        **kwargs,
    ):
        from ..integrations import get_keys_to_not_convert, replace_with_bitnet_linear

        self.modules_to_not_convert = get_keys_to_not_convert(model)

        if self.quantization_config.modules_to_not_convert is not None:
            self.modules_to_not_convert.extend(self.quantization_config.modules_to_not_convert)

        model = replace_with_bitnet_linear(
            model,
            modules_to_not_convert=self.modules_to_not_convert,
            quantization_config=self.quantization_config,
            pre_quantized=self.pre_quantized,
        )

    def adjust_max_memory(self, max_memory: Dict[str, Union[int, str]]) -> Dict[str, Union[int, str]]:
        max_memory = {key: val * 0.90 for key, val in max_memory.items()}
        return max_memory

    def adjust_target_dtype(self, target_dtype: "torch.dtype") -> "torch.dtype":
        target_dtype = torch.int8
        return target_dtype

    def is_serializable(self, safe_serialization=None):
        return True

    @property
    def is_trainable(self) -> bool:


SUMMARY:
**Class Summary: BitNetHfQuantizer**

**Purpose and Functionality**
=========================

The `BitNetHfQuantizer` class implements 1.58-bit quantization from the BitNet method, primarily for use with Hugging Face's transformers. It converts linear layers into BitLinear layers before loading and requires calibration.

**Key Implementation Details**
-----------------------------

*   Converts linear layers to BitLinear layers during model loading
*   Uses `accelerate` package for dependency
*   Raises errors if necessary dependencies (`accumulate`) are not installed or specific device maps contain CPU or disk devices

**Method Signatures and Interfaces**
------------------------------------

*   `validate_env`: checks environment availability, device usage, and model format support
*   `_process_model_before_WEIGHT_LOADING`: processes model before weight loading, converting certain modules to BitLinear layers
*   `_process_model_after_WEIGHT Loading`: an empty method for future implementation

**Dependencies and Requirements**
-------------------------------

*   Requires `accelerate` package
*   Supports ternary weights in PyTorch format (though loading from TF or FLAX is currently not supported)

**Notable Dependencies**

*   `torch`
*   `accumulate` (via `accelerate`)
*   `get_keys_to_not_convert`, `replace_with_bitnet_linear` from `.integrations` module (not shown in this code snippet)


key: chroma:document
string_value:
CODE:

class Bnb4BitHfQuantizer(HfQuantizer):
    """
    4-bit quantization from bitsandbytes.py quantization method:
        before loading: converts transformer layers into Linear4bit during loading: load 16bit weight and pass to the
        layer object after: quantizes individual weights in Linear4bit into 4bit at the first .cuda() call
        saving:
            from state dict, as usual; saves weights and `quant_state` components
        loading:
            need to locate `quant_state` components and pass to Param4bit constructor
    """

    use_keep_in_fp32_modules = True
    requires_parameters_quantization = True
    requires_calibration = False

    required_packages = ["bitsandbytes", "accelerate"]

    def __init__(self, quantization_config, **kwargs):
        super().__init__(quantization_config, **kwargs)

        if self.quantization_config.llm_int8_skip_modules is not None:
            self.modules_to_not_convert = self.quantization_config.llm_int8_skip_modules

    def validate_environment(self, *args, **kwargs):
        if not is_accelerate_available():
            raise ImportError(
                f"Using `bitsandbytes` 4-bit quantization requires Accelerate: `pip install 'accelerate>={ACCELERATE_MIN_VERSION}'`"
            )
        if not is_bitsandbytes_available():
            raise ImportError(
                "Using `bitsandbytes` 4-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes`"
            )

        from ..integrations import validate_bnb_backend_availability
        from ..utils import is_bitsandbytes_multi_backend_available

        bnb_multibackend_is_enabled = is_bitsandbytes_multi_backend_available()
        validate_bnb_backend_availability(raise_exception=True)

        if kwargs.get("from_tf", False) or kwargs.get("from_flax", False):
            raise ValueError(
                "Converting into 4-bit or 8-bit weights from tf/flax weights is currently not supported, please make"
                " sure the weights are in PyTorch format."
            )

        device_map = kwargs.get("device_map", None)
        if (
            device_map is not None
            and isinstance(device_map, dict)
            and not self.quantization_config.llm_int8_enable_fp32_cpu_offload
        ):
            device_map_without_lm_head = {
                key: device_map[key] for key in device_map.keys() if key not in self.modules_to_not_convert
            }
            if set(device_map.values()) == {"cpu"} and bnb_multibackend_is_enabled:
                pass
            elif "cpu" in device_map_without_lm_head.values() or "disk" in device_map_without_lm_head.values():
                raise ValueError(
                    "Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the "
                    "quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules "
                    "in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to "
                    "`from_pretrained`. Check "
                    "https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu "
                    "for more details. "
                )

        if version.parse(importlib.metadata.version("bitsandbytes")) < version.parse("0.39.0"):
            raise ValueError(
                "You have a version of `bitsandbytes` that is not compatible with 4bit inference and training"
                " make sure you have the latest version of `bitsandbytes` installed"
            )

    def adjust_target_dtype(self, target_dtype: "torch.dtype") -> "torch.dtype":
        if version.parse(importlib.metadata.version("accelerate")) > version.parse("0.19.0"):
            from accelerate.utils import CustomDtype

            if target_dtype != torch.int8:
                logger.info("target_dtype {target_dtype} is replaced by `CustomDtype.INT4` for 4-bit BnB quantization")
            return CustomDtype.INT4
        else:
            raise ValueError(
                "You are using `device_map='auto'` on a 4bit loaded version of the model. To automatically compute"
                " the appropriate device map, you should upgrade your `accelerate` library,"
                "`pip install --upgrade accelerate` or install it from source to support fp4 auto device map"
                "calculation. You may encounter unexpected behavior, or pass your own device map"
            )

    def check_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        state_dict: Dict[str, Any],
        **kwargs,
    ) -> bool:
        import bitsandbytes as bnb

        module, tensor_name = get_module_from_name(model, param_name)
        if isinstance(module._parameters.get(tensor_name, None), bnb.nn.Params4bit):
            # Add here check for loaded components' dtypes once serialization is implemented
            return True
        elif isinstance(module, bnb.nn.Linear4bit) and tensor_name == "bias":
            # bias could be loaded by regular set_module_tensor_to_device() from accelerate,
            # but it would wrongly use uninitialized weight there.
            return True
        else:
            return False

    def create_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        target_device: "torch.device",
        state_dict: Dict[str, Any],
        unexpected_keys: Optional[List[str]] = None,
    ):
        """
        combines logic from _load_state_dict_into_meta_model and .integrations.bitsandbytes.py::set_module_quantized_tensor_to_device()
        """
        import bitsandbytes as bnb

        module, tensor_name = get_module_from_name(model, param_name)

        if tensor_name not in module._parameters:
            raise ValueError(f"{module} does not have a parameter or a buffer named {tensor_name}.")

        old_value = getattr(module, tensor_name)

        # `torch.Tensor.to(<int num>)` is not supported by `torch_npu` (see this [issue](https://github.com/Ascend/pytorch/issues/16)).
        if isinstance(target_device, int) and is_torch_npu_available():
            target_device = f"npu:{target_device}"
        if tensor_name == "bias":
            if param_value is None:
                new_value = old_value.to(target_device)
            else:
                new_value = param_value.to(target_device)

            new_value = torch.nn.Parameter(new_value, requires_grad=old_value.requires_grad)
            module._parameters[tensor_name] = new_value
            return

        if not isinstance(module._parameters[tensor_name], bnb.nn.Params4bit):
            raise ValueError("this function only loads `Linear4bit components`")
        if (
            old_value.device == torch.device("meta")
            and target_device not in ["meta", torch.device("meta")]
            and param_value is None
        ):
            raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {target_device}.")

        # construct `new_value` for the module._parameters[tensor_name]:
        if self.pre_quantized:
            # 4bit loading. Collecting components for restoring quantized weight
            # This can be expanded to make a universal call for any quantized weight loading

            if not self.is_serializable:
                raise ValueError(
                    "Detected int4 weights but the version of bitsandbytes is not compatible with int4 serialization. "
                    "Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`."
                )

            if (param_name + ".quant_state.bitsandbytes__fp4" not in state_dict) and (
                param_name + ".quant_state.bitsandbytes__nf4" not in state_dict
            ):
                raise ValueError(
                    f"Supplied state dict for {param_name} does not contain `bitsandbytes__*` and possibly other `quantized_stats` components."
                )

            quantized_stats = {}
            for k, v in state_dict.items():
                if param_name + "." in k:
                    quantized_stats[k] = v
                    if unexpected_keys is not None and k in unexpected_keys:
                        unexpected_keys.remove(k)

            param_kwargs = {}
            if self.is_bnb_supports_quant_storage_module:
                param_kwargs["module"] = module

            new_value = bnb.nn.Params4bit.from_prequantized(
                data=param_value,
                quantized_stats=quantized_stats,
                requires_grad=False,
                device=target_device,
                **param_kwargs,
            )
        else:
            new_value = param_value.to("cpu")

            # Support models using `Conv1D` in place of `nn.Linear` (e.g. openai-community/gpt2) by transposing the weight matrix prior to quantization.
            # Since weights are saved in the correct "orientation", we skip transposing when loading.
            if issubclass(module.source_cls, Conv1D):
                new_value = new_value.T

            kwargs = old_value.__dict__
            new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).to(target_device)

        module._parameters[tensor_name] = new_value

    # Copied from transformers.quantizers.quantizer_bnb_8bit.Bnb8BitHfQuantizer.adjust_max_memory
    def adjust_max_memory(self, max_memory: Dict[str, Union[int, str]]) -> Dict[str, Union[int, str]]:
        # need more space for buffers that are created during quantization
        max_memory = {key: val * 0.90 for key, val in max_memory.items()}
        return max_memory

    # Copied from transformers.quantizers.quantizer_bnb_8bit.Bnb8BitHfQuantizer.update_torch_dtype
    def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
        if torch_dtype is None:
            # We force the `dtype` to be float16, this is a requirement from `bitsandbytes`
            logger.info(
                "Overriding torch_dtype=%s with `torch_dtype=torch.float16` due to "
                "requirements of `bitsandbytes` to enable model loading in 8-bit or 4-bit. "
                "Pass your own torch_dtype to specify the dtype of the remaining non-linear layers or pass"
                " torch_dtype=torch.float16 to remove this warning.",
                torch_dtype,
            )
            torch_dtype = torch.float16
        return torch_dtype

    def update_device_map(self, device_map):
        if device_map is None:
            if torch.cuda.is_available():
                device_map = {"": torch.cuda.current_device()}
            elif is_torch_npu_available():
                device_map = {"": f"npu:{torch.npu.current_device()}"}
            elif is_torch_xpu_available():
                device_map = {"": f"xpu:{torch.xpu.current_device()}"}
            else:
                device_map = {"": "cpu"}
            logger.info(
                "The device_map was not initialized. "
                f"Setting device_map to {device_map}. "
                "If you want to use the model for inference, please set device_map ='auto' "
            )
        return device_map

    # Copied from transformers.quantizers.quantizer_bnb_8bit.Bnb8BitHfQuantizer._process_model_before_weight_loading
    def _process_model_before_weight_loading(
        self,
        model: "PreTrainedModel",
        device_map,
        keep_in_fp32_modules: List[str] = [],
        **kwargs,
    ):
        from ..integrations import get_keys_to_not_convert, replace_with_bnb_linear

        llm_int8_enable_fp32_cpu_offload = self.quantization_config.llm_int8_enable_fp32_cpu_offload

        # We keep some modules such as the lm_head in their original dtype for numerical stability reasons
        if self.quantization_config.llm_int8_skip_modules is None:
            self.modules_to_not_convert = get_keys_to_not_convert(model)
        else:
            self.modules_to_not_convert = self.quantization_config.llm_int8_skip_modules

        if not isinstance(self.modules_to_not_convert, list):
            self.modules_to_not_convert = [self.modules_to_not_convert]

        self.modules_to_not_convert.extend(keep_in_fp32_modules)

        # Extend `self.modules_to_not_convert` to keys that are supposed to be offloaded to `cpu` or `disk`
        if isinstance(device_map, dict) and len(device_map.keys()) > 1:
            keys_on_cpu = [key for key, value in device_map.items() if value in ["disk", "cpu"]]

            if len(keys_on_cpu) > 0 and not llm_int8_enable_fp32_cpu_offload:
                raise ValueError(
                    "If you want to offload some keys to `cpu` or `disk`, you need to set "
                    "`llm_int8_enable_fp32_cpu_offload=True`. Note that these modules will not be "
                    " converted to 8-bit but kept in 32-bit."
                )
            self.modules_to_not_convert.extend(keys_on_cpu)

        model = replace_with_bnb_linear(
            model, modules_to_not_convert=self.modules_to_not_convert, quantization_config=self.quantization_config
        )
        # TODO: consider bringing replace_with_bnb_linear() code from ..integrations/bitsandbyter.py to here

        model.config.quantization_config = self.quantization_config

    # Copied from transformers.quantizers.quantizer_bnb_8bit.Bnb8BitHfQuantizer._process_model_after_weight_loading with 8bit->4bit
    def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
        model.is_loaded_in_4bit = True
        model.is_4bit_serializable = self.is_serializable()
        return model

    def is_serializable(self, safe_serialization=None):
        _is_4bit_serializable = version.parse(importlib.metadata.version("bitsandbytes")) >= version.parse("0.41.3")

        if not _is_4bit_serializable:
            logger.warning(
                "You are calling `save_pretrained` to a 4-bit converted model, but your `bitsandbytes` version doesn't support it. "
                "If you want to save 4-bit models, make sure to have `bitsandbytes>=0.41.3` installed."
            )
            return False

        return True

    @cached_property
    def is_bnb_supports_quant_storage_module(self) -> bool:
        """
        determines if the current version of bitsandbytes supports
        the `module` parameter in `Params4bit.from_prequantized`
        :return:
        """
        return version.parse(importlib.metadata.version("bitsandbytes")) >= version.parse("0.43.3")

    @property
    def is_trainable(self) -> bool:
        return True

    def _dequantize(self, model):
        from ..integrations import dequantize_and_replace

        model = dequantize_and_replace(
            model, self.modules_to_not_convert, quantization_config=self.quantization_config
        )


SUMMARY:
This is the code for a class in the Hugging Face Transformers library, specifically designed for quantizing models. The class `Bnb8BitHfQuantizer` inherits from `BaseQuantizer` and provides functionality for converting pre-trained models into 8-bit quantized versions using the BNB (Binary Neural Network) format.

Here's a concise summary of the class's responsibilities:

1. **Device mapping**: It initializes or updates the device map to determine where each model parameter should be stored when running on different devices.
2. **Model processing**: Before loading model weights, it processes the model by replacing some modules with BNB-compatible linear layers and configuring the quantization settings for the model's architecture.
3. **Weight loading and conversion**: It loads the pre-trained model weights and converts them into 8-bit quantized values using the `Params4bit.from_prequantized` method from BitBox.
4. **Dequantizing**: When running inference, it can dequantize the converted 8-bit BNB models back to their original float32 values before passing them through the neural network.

The class is designed to support both training and inference workflows with quantization. It uses a configuration-driven approach, where users specify how the model should be processed (e.g., which modules not to convert, whether to enable llm-int8 features) using pre-defined configuration options.

Overall, this class simplifies the process of converting pre-trained Hugging Face models into 8-bit quantized BNB versions with minimal additional code and effort required from users.


key: chroma:document
string_value:
CODE:

def get_num_sms_from_device(device):
    target_device_cc = torch.cuda.get_device_capability(device=device)
    if target_device_cc == (8, 6):
        return 84
    elif target_device_cc == (8, 0):
        return 108
    elif target_device_cc == (8, 9):
        return 128
    else:
        raise NotImplementedError(
            f"Device capability {target_device_cc} not supported for FLUTE (yet?) to verify your device capability check out https://developer.nvidia.com/cuda-gpus"


SUMMARY:
Here is a concise summary of the provided function code chunk:

**Function Summary: `get_num_sms_from_device(device)`**

Determines the number of SMS-like messages that can be processed by the specified CUDA device. 

* Returns different values based on the device's capability:
	+ 84 for devices with capability `(8, 6)`
	+ 108 for devices with capability `(8, 0)`
	+ 128 for devices with capability `(8, 9)`
* Raises a `NotImplementedError` if an unsupported capability is encountered.
 

External Dependency: 
- `torch.cuda.get_device_capability(device=device)` 
- `torch.` ( Torch library)

Key Implementation Details: 
- Device capabilities and corresponding SMS-like message counts are hardcoded.


key: chroma:document
string_value:
CODE:

class HiggsHfQuantizer(HfQuantizer):
    """
    Quantizer of the HIGGS method. Enables the loading of prequantized models and in-flight quantization of full-precision models.
    """

    requires_calibration = False
    requires_parameters_quantization = True
    required_packages = ["flute-kernel", "fast_hadamard_transform"]

    def __init__(self, quantization_config: QuantizationConfigMixin, **kwargs):
        super().__init__(quantization_config, **kwargs)
        self.quantization_config = quantization_config

    def validate_environment(self, device_map, **kwargs):
        if not torch.cuda.is_available():
            raise NotImplementedError("HIGGS quantization is only supported on GPU. Please use a different quantizer.")

        if not is_accelerate_available():
            raise ImportError("Using `higgs` quantization requires Accelerate: `pip install accelerate`")

        if not is_flute_available():
            raise ImportError("Using `higgs` quantization requires FLUTE: `pip install flute-kernel>=0.3.0`")

        if not is_hadamard_available():
            raise ImportError(
                "Using `higgs` quantization requires fast_hadamard_transform: `pip install fast_hadamard_transform`"
            )

        if device_map is None:
            raise ValueError(
                "You are attempting to load a HIGGS model without setting device_map."
                " Please set device_map comprised of 'cuda' devices."
            )
        elif isinstance(device_map, dict) and ("cpu" in device_map.values() or "disk" in device_map.values()):
            raise ValueError(
                "You are attempting to load a HIGGS model with a device_map that contains a CPU or disk device."
                " This is not supported. Please remove the CPU or disk device from the device_map."
            )

    def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
        if torch_dtype is None:
            logger.info("`torch_dtype` is None. Setting `torch_dtype=torch.float16` for FLUTE compatibility.")
            torch_dtype = torch.float16
        elif torch_dtype != torch.float16 and torch_dtype != torch.bfloat16:
            raise ValueError(
                f"Invalid `torch_dtype` {torch_dtype}. HIGGS quantization only supports `torch_dtype=torch.float16` or `torch_dtype=torch.bfloat16`."
            )

        return torch_dtype

    def create_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        target_device: "torch.device",
        state_dict: Dict[str, Any],
        unexpected_keys: Optional[List[str]] = None,
    ):
        from ..integrations import quantize_with_higgs

        """
        Quantizes weights into weight and weight_scale
        """
        flute_dict = quantize_with_higgs(
            param_value.to(target_device),
            self.quantization_config.bits,
            self.quantization_config.p,
            self.quantization_config.group_size,
            self.quantization_config.hadamard_size,
        )

        del param_value

        module, tensor_name = get_module_from_name(model, param_name)
        for key, value in flute_dict.items():
            if key in module._parameters:
                module._parameters[key] = torch.nn.Parameter(value, requires_grad=False)
            elif key in module._buffers:
                module._buffers[key] = torch.nn.Buffer(value)
            else:
                raise ValueError(f"Unexpected key {key} in module {module}")

        if unexpected_keys is not None and param_name in unexpected_keys:
            unexpected_keys.remove(param_name)

        module.num_sms_packed = torch.nn.Parameter(
            torch.tensor(get_num_sms_from_device(target_device), device=target_device, dtype=torch.int32),
            requires_grad=False,
        )

    def _process_model_before_weight_loading(
        self,
        model: "PreTrainedModel",
        **kwargs,
    ):
        from ..integrations import replace_with_higgs_linear

        replace_with_higgs_linear(
            model,
            quantization_config=self.quantization_config,
        )
        model.config.quantization_config = self.quantization_config

    def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
        import flute.utils

        from ..integrations import HiggsLinear

        flute_workspaces = {}
        for name, module in model.named_modules():
            if isinstance(module, HiggsLinear):
                # Every HiggsLinear needs a "workspace": a buffer for the unpacking operation.
                # This buffer needs to be on the same device as the weights, but can be reused across modules otherwise.
                if module.weight.device not in flute_workspaces:
                    flute_workspaces[module.weight.device] = flute.utils.make_workspace_streamk(
                        device=module.weight.device
                    )
                module.workspace = flute_workspaces[module.weight.device]

                # FLUTE weights are packed in a way that is optimized for a specific number of SMs (GPU streaming multiprocessors).
                # If the model is loaded on a different device than the one it was saved on, we need to repack the weights.
                if module.num_sms_packed.item() != get_num_sms_from_device(module.weight.device):
                    new_device = module.weight.device
                    new_num_sms = get_num_sms_from_device(new_device)
                    module.weight.data = flute.utils.pack(
                        flute.utils.unpack(
                            weight=module.weight.data,
                            scales=module.scales.data,
                            workspace=module.workspace,
                            num_bits=module.num_bits,
                            group_size=module.group_size,
                            num_sms_packed=module.num_sms_packed.item(),
                        ).T.contiguous(),
                        module.num_bits,
                        module.group_size,
                    )
                    module.num_sms_packed = torch.nn.Parameter(
                        torch.tensor(new_num_sms, device=new_device, dtype=torch.int32),
                        requires_grad=False,
                    )

    def update_missing_keys(self, model, missing_keys: List[str], prefix: str) -> List[str]:
        from ..integrations import HiggsLinear

        not_missing_keys = []
        for name, module in model.named_modules():
            if isinstance(module, HiggsLinear):
                for missing in missing_keys:
                    if (
                        (name in missing or name in f"{prefix}.{missing}")
                        and not missing.endswith(".weight")
                        and not missing.endswith(".bias")
                    ):
                        not_missing_keys.append(missing)
        return [k for k in missing_keys if k not in not_missing_keys]

    @property
    def is_trainable(self, model: Optional["PreTrainedModel"] = None):
        return False

    def is_serializable(self, safe_serialization=None):
        return True

    def check_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        state_dict: Dict[str, Any],
        **kwargs,
    ) -> bool:
        from ..integrations import HiggsLinear

        module, tensor_name = get_module_from_name(model, param_name)
        if isinstance(module, HiggsLinear) and tensor_name == "weight" and param_value.dtype != torch.int16:
            # Only quantize weights of HiggsLinear modules that are not already quantized
            return True
        else:
            return False

    def _dequantize(self, model):
        from ..integrations import dequantize_higgs

        model = dequantize_higgs(model)


SUMMARY:
**Class Summary: HiggsHfQuantizer**

The `HiggsHfQuantizer` class is a quantization method for full-precision models. It enables in-flight quantization and loading of pre-quantized models.

**Primary Purpose**
- Enables the loading of pre-quantized models
- In-flight quantization of full-precision models

**Key Implementation Details**

*   `create_quantized_param`: Quantizes weights into `weight` and `weight_scale`
*   `_process_model_before_weight_loading`: Replaces linear layers with HiggsLinear modules for calibration
*   `_process_model_after_weight_loading`: Repacks FLUTE weights if necessary
*   `update_torch_dtype`: Updates PyTorch data type to support FLUTE compatibility
*   `_dequantize`: De-quantizes the model using the HIGGS method

**Notable Dependencies and Requirements**

*   `flute-kernel`
*   `fast_hadamard_transform`
*   Accelerate (`pip install accelerate`)
*   PyTorch (`torch`)

This class provides a quantization method for full-precision models, enabling efficient loading of pre-quantized models and in-flight quantization. The implementation details focus on key methods such as `create_quantized_param`, `_process_model_before_weight_loading`, and `_update_torch_dtype`.


key: chroma:document
string_value:
CODE:

class EetqHfQuantizer(HfQuantizer):
    """
    8-bit quantization from EETQ quantization method:
        before loading: converts transformer layers into W8A16Linear during loading: load 16bit weight and pass to the
        layer object after: quantizes individual weights in Linear8bitLt into 8bit at first .cuda() call
    """

    requires_parameters_quantization = True
    requires_calibration = False

    required_packages = ["eetq", "accelerate"]

    def __init__(self, quantization_config, **kwargs):
        super().__init__(quantization_config, **kwargs)
        self.quantization_config = quantization_config

    def validate_environment(self, *args, **kwargs):
        if not is_eetq_available():
            raise ImportError(
                "Using `eetq` 8-bit quantization requires eetq."
                "Please install the latest version of eetq from : https://github.com/NetEase-FuXi/EETQ"
            )

        try:
            import eetq  # noqa: F401
        except ImportError as exc:
            if "shard_checkpoint" in str(exc):
                # EETQ 1.0.0 is currently broken with the latest transformers because it tries to import the removed
                # shard_checkpoint function, see https://github.com/NetEase-FuXi/EETQ/issues/34.
                # TODO: Update message once eetq releases a fix
                raise ImportError(
                    "You are using a version of EETQ that is incompatible with the current transformers version. "
                    "Either downgrade transformers to <= v4.46.3 or, if available, upgrade EETQ to > v1.0.0."
                ) from exc
            else:
                raise

        if not is_accelerate_available():
            raise ImportError("Loading an EETQ quantized model requires accelerate (`pip install accelerate`)")

        if kwargs.get("from_tf", False) or kwargs.get("from_flax", False):
            raise ValueError(
                "Converting into 8-bit weights from tf/flax weights is currently not supported, please make"
                " sure the weights are in PyTorch format."
            )

        if not torch.cuda.is_available():
            raise RuntimeError("No GPU found. A GPU is needed for quantization.")

        device_map = kwargs.get("device_map", None)
        if device_map is None:
            logger.warning_once(
                "You have loaded an EETQ model on CPU and have a CUDA device available, make sure to set "
                "your model on a GPU device in order to run your model."
            )
        elif device_map is not None:
            if isinstance(device_map, dict) and ("cpu" in device_map.values() or "disk" in device_map.values()):
                raise ValueError(
                    "You are attempting to load an EETQ model with a device_map that contains a CPU or disk device."
                    " This is not supported. Please remove the CPU or disk device from the device_map."
                )

    def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
        if torch_dtype is None:
            torch_dtype = torch.float16
            logger.info(
                "Overriding torch_dtype=%s with `torch_dtype=torch.float16` due to "
                "requirements of `eetq` to enable model loading in 8-bit. "
                "Pass your own torch_dtype to specify the dtype of the remaining non-linear layers or pass"
                " torch_dtype=torch.float16 to remove this warning.",
                torch_dtype,
            )
        elif torch_dtype != torch.float16:
            logger.info("We suggest you to set `torch_dtype=torch.float16` for better efficiency with EETQ.")
        return torch_dtype

    def check_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        state_dict: Dict[str, Any],
        **kwargs,
    ):
        from eetq import EetqLinear

        module, tensor_name = get_module_from_name(model, param_name)

        if isinstance(module, EetqLinear):
            if self.pre_quantized or tensor_name == "bias":
                if tensor_name == "weight" and param_value.dtype != torch.int8:
                    raise ValueError("Expect quantized weights but got an unquantized weight")
                return False
            else:
                if tensor_name == "weight_scale":
                    raise ValueError("Expect unquantized weights but got a quantized weight_scale")
                return True
        return False

    def create_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        target_device: "torch.device",
        state_dict: Dict[str, Any],
        unexpected_keys: Optional[List[str]] = None,
    ):
        """
        quantizes weights into qweight and weight_scales
        """
        from eetq import quantize_and_preprocess_weights

        module, tensor_name = get_module_from_name(model, param_name)
        new_value, weight_scale = quantize_and_preprocess_weights(param_value)

        module._buffers[tensor_name] = new_value.to(target_device)
        module.register("weight_scales", weight_scale.to(target_device))

    def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
        return model

    def _process_model_before_weight_loading(
        self,
        model: "PreTrainedModel",
        device_map,
        keep_in_fp32_modules: List[str] = [],
        **kwargs,
    ):
        from ..integrations import get_keys_to_not_convert, replace_with_eetq_linear

        self.modules_to_not_convert = get_keys_to_not_convert(model)

        if self.quantization_config.modules_to_not_convert is not None:
            self.modules_to_not_convert.extend(self.quantization_config.modules_to_not_convert)

        model = replace_with_eetq_linear(
            model,
            modules_to_not_convert=self.modules_to_not_convert,
            quantization_config=self.quantization_config,
            pre_quantized=self.pre_quantized,
        )

        model.config.quantization_config = self.quantization_config

    def is_serializable(self, safe_serialization=None):
        return True

    @property
    def is_trainable(self) -> bool:


SUMMARY:
**EetqHfQuantizer Class Summary**

This class extends HfQuantizer and implements 8-bit quantization using the EETQ (NetEase Transformers Enhanced Quantization) method.

**Purpose**
The primary purpose of this class is to convert transformer layers into W8A16Linear, load 16bit weights, and then quantize individual weights in Linear8bitLt into 8bit at first use on a CUDA device.

**Key Implementation Details**

* `update_torch_dtype`: Updates the torch.dtype to tensor.float16 if not specified.
* `check_quantized_param` and `create_quantized_param`: Check and create quantized parameters for a model.
* `_process_model_after_weight_loading` and `_process_model_before_weight_loading`: Process models before and after loading weights.

**Notable Dependencies**

* `eetq`: Required package for EETQ quantization method
* `accelerate`: Required package for CUDA functionality
* `transformers`: Used module for getting tensor names

**Requirements**

* The model must be compatible with the current transformers version (<= v4.46.3).
* The weights must be in PyTorch format.
* A GPU is required for quantization.
* The device map must not contain a CPU or disk device.

**Methods**

* `__init__`: Initializes the EetqHfQuantizer with quantization_config.
* `validate_environment`: Validates the environment before loading the model.
* `is_serializable` and `is_trainable`: Always returns True.


key: chroma:document
string_value:
CODE:

class GptqHfQuantizer(HfQuantizer):
    """
    Quantizer of the GPTQ method - for GPTQ the quantizer support calibration of the model through
    `auto_gptq` package. Quantization is done under the hood for users if they load a non-prequantized model.
    """

    requires_calibration = False
    required_packages = ["optimum", "auto_gptq"]
    optimum_quantizer = None

    def __init__(self, quantization_config: QuantizationConfigMixin, **kwargs):
        super().__init__(quantization_config, **kwargs)

        if not is_optimum_available():
            raise ImportError("Loading a GPTQ quantized model requires optimum (`pip install optimum`)")
        from optimum.gptq import GPTQQuantizer

        self.optimum_quantizer = GPTQQuantizer.from_dict(self.quantization_config.to_dict_optimum())

    def validate_environment(self, *args, **kwargs):
        if not is_optimum_available():
            raise ImportError("Loading a GPTQ quantized model requires optimum (`pip install optimum`)")

        if not is_auto_gptq_available():
            raise ImportError(
                "Loading a GPTQ quantized model requires the auto-gptq library (`pip install auto-gptq`)"
            )

        gptq_supports_cpu = version.parse(importlib.metadata.version("auto-gptq")) > version.parse("0.4.2")
        if not gptq_supports_cpu and not torch.cuda.is_available():
            raise RuntimeError("GPU is required to quantize or run quantize model.")
        elif version.parse(importlib.metadata.version("auto_gptq")) < version.parse("0.4.2"):
            raise ImportError(
                "You need a version of auto_gptq >= 0.4.2 to use GPTQ: `pip install --upgrade auto-gptq`"
            )

    def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
        if torch_dtype is None:
            torch_dtype = torch.float16
            logger.info("Loading the model in `torch.float16`. To overwrite it, set `torch_dtype` manually.")
        elif torch_dtype != torch.float16:
            logger.info("We suggest you to set `torch_dtype=torch.float16` for better efficiency with GPTQ.")
        return torch_dtype

    def _process_model_before_weight_loading(self, model: "PreTrainedModel", **kwargs):
        if model.__class__.main_input_name != "input_ids":
            raise RuntimeError("We can only quantize pure text model.")

        if self.pre_quantized:
            model = self.optimum_quantizer.convert_model(model)

    def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
        if self.pre_quantized:
            model = self.optimum_quantizer.post_init_model(model)
        else:
            if self.quantization_config.tokenizer is None:
                self.quantization_config.tokenizer = model.name_or_path

            self.optimum_quantizer.quantize_model(model, self.quantization_config.tokenizer)
            model.config.quantization_config = GPTQConfig.from_dict(self.optimum_quantizer.to_dict())

    @property
    def is_trainable(self, model: Optional["PreTrainedModel"] = None):
        return True

    def is_serializable(self, safe_serialization=None):


SUMMARY:
**Class Summary**

`GptqHfQuantizer` - Quantizer of the GPTQ method
=====================================================

### Main Purpose and Functionality

This class is responsible for quantizing models using the GPTQ (Gradient-based Proportionate Quantization) method. It requires calibration through the `auto_gptq` package.

### Key Implementation Details

* Uses the `optimum` library to perform quantization.
* Validates environment dependencies, including `auto-gptq` and `torch`.
* Provides methods for updating PyTorch data types and processing models before weight loading and after.

### Notable Dependencies and Requirements

* `optimum`
* `auto_gptq`
* `torch`

### Method Signatures and Interfaces

* `validate_environment()`: Validates environment dependencies.
* `_process_model_before_weight_loading(model)`: Processes model before weight loading.
* `_process_model_after_weight_loading(model)`: Processes model after weight loading.
* `update_torch_dtype(torch_dtype) -> torch.dtype`: Updates PyTorch data type.

### Configuration and Inheritance

Inherits from `HfQuantizer`. Requires calibration set to `False`.


key: chroma:document
string_value:
CODE:

def get_module_from_name(module, tensor_name: str) -> Tuple[Any, str]:
    if "." in tensor_name:
        splits = tensor_name.split(".")
        for split in splits[:-1]:
            new_module = getattr(module, split)
            if new_module is None:
                raise ValueError(f"{module} has no attribute {split}.")
            module = new_module
        tensor_name = splits[-1]


SUMMARY:
**Summary:**

The `get_module_from_name` function retrieves a nested module from a given module path. It takes a parent module (`module`) and a tensor name (with optional path separators) as input and returns the corresponding module object and original tensor name.

**Key Details:**

*   Uses `getattr` to dynamically traverse attribute hierarchy.
*   Iterates through dot-separated path segments, assigning each intermediate sub-module.
*   Raises `ValueError` if an invalid attribute is not found.

**Return Values:**

*   `module`: the nested module object
*   `tensor_name`: the original tensor name (possibly modified)

**Notable Dependencies:**

*   `module` - parent module or module path prefix
*   `getattr` - used for dynamic attribute access.


key: chroma:document
string_value:
CODE:

class Bnb8BitHfQuantizer(HfQuantizer):
    """
    8-bit quantization from bitsandbytes quantization method:
        before loading: converts transformer layers into Linear8bitLt during loading: load 16bit weight and pass to the
        layer object after: quantizes individual weights in Linear8bitLt into 8bit at fitst .cuda() call
    saving:
        from state dict, as usual; saves weights and 'SCB' component
    loading:
        need to locate SCB component and pass to the Linear8bitLt object
    """

    use_keep_in_fp32_modules = True
    requires_parameters_quantization = True
    requires_calibration = False

    required_packages = ["bitsandbytes", "accelerate"]

    def __init__(self, quantization_config, **kwargs):
        super().__init__(quantization_config, **kwargs)

        if self.quantization_config.llm_int8_skip_modules is not None:
            self.modules_to_not_convert = self.quantization_config.llm_int8_skip_modules

    def validate_environment(self, *args, **kwargs):
        if not is_accelerate_available():
            raise ImportError(
                f"Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install 'accelerate>={ACCELERATE_MIN_VERSION}'`"
            )
        if not is_bitsandbytes_available():
            raise ImportError(
                "Using `bitsandbytes` 8-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes`"
            )

        from ..integrations import validate_bnb_backend_availability
        from ..utils import is_bitsandbytes_multi_backend_available

        bnb_multibackend_is_enabled = is_bitsandbytes_multi_backend_available()
        validate_bnb_backend_availability(raise_exception=True)

        if kwargs.get("from_tf", False) or kwargs.get("from_flax", False):
            raise ValueError(
                "Converting into 4-bit or 8-bit weights from tf/flax weights is currently not supported, please make"
                " sure the weights are in PyTorch format."
            )

        device_map = kwargs.get("device_map", None)
        if (
            device_map is not None
            and isinstance(device_map, dict)
            and not self.quantization_config.llm_int8_enable_fp32_cpu_offload
        ):
            device_map_without_lm_head = {
                key: device_map[key] for key in device_map.keys() if key not in self.modules_to_not_convert
            }
            if set(device_map.values()) == {"cpu"} and bnb_multibackend_is_enabled:
                pass
            elif "cpu" in device_map_without_lm_head.values() or "disk" in device_map_without_lm_head.values():
                raise ValueError(
                    "Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the "
                    "quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules "
                    "in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to "
                    "`from_pretrained`. Check "
                    "https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu "
                    "for more details. "
                )

        if version.parse(importlib.metadata.version("bitsandbytes")) < version.parse("0.37.2"):
            raise ValueError(
                "You have a version of `bitsandbytes` that is not compatible with 8bit inference and training"
                " make sure you have the latest version of `bitsandbytes` installed"
            )

    def adjust_max_memory(self, max_memory: Dict[str, Union[int, str]]) -> Dict[str, Union[int, str]]:
        # need more space for buffers that are created during quantization
        max_memory = {key: val * 0.90 for key, val in max_memory.items()}
        return max_memory

    def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
        if torch_dtype is None:
            # We force the `dtype` to be float16, this is a requirement from `bitsandbytes`
            logger.info(
                "Overriding torch_dtype=%s with `torch_dtype=torch.float16` due to "
                "requirements of `bitsandbytes` to enable model loading in 8-bit or 4-bit. "
                "Pass your own torch_dtype to specify the dtype of the remaining non-linear layers or pass"
                " torch_dtype=torch.float16 to remove this warning.",
                torch_dtype,
            )
            torch_dtype = torch.float16
        return torch_dtype

    def update_device_map(self, device_map):
        if device_map is None:
            if torch.cuda.is_available():
                device_map = {"": torch.cuda.current_device()}
            elif is_torch_xpu_available():
                device_map = {"": f"xpu:{torch.xpu.current_device()}"}
            else:
                device_map = {"": "cpu"}
            logger.info(
                "The device_map was not initialized. "
                f"Setting device_map to {device_map}. "
                "If you want to use the model for inference, please set device_map ='auto' "
            )
        return device_map

    def adjust_target_dtype(self, target_dtype: "torch.dtype") -> "torch.dtype":
        if target_dtype != torch.int8:
            logger.info("target_dtype {target_dtype} is replaced by `torch.int8` for 8-bit BnB quantization")
        return torch.int8

    def check_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        state_dict: Dict[str, Any],
        **kwargs,
    ):
        import bitsandbytes as bnb

        module, tensor_name = get_module_from_name(model, param_name)
        if isinstance(module._parameters.get(tensor_name, None), bnb.nn.Int8Params):
            if self.pre_quantized:
                if param_name.replace("weight", "SCB") not in state_dict.keys():
                    raise ValueError("Missing quantization component `SCB`")
                if param_value.dtype != torch.int8:
                    raise ValueError(
                        f"Incompatible dtype `{param_value.dtype}` when loading 8-bit prequantized weight. Expected `torch.int8`."
                    )
            return True
        return False

    def create_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        target_device: "torch.device",
        state_dict: Dict[str, Any],
        unexpected_keys: Optional[List[str]] = None,
    ):
        """
        combines logic from _load_state_dict_into_meta_model and .integrations.bitsandbytes.py::set_module_quantized_tensor_to_device()
        needs aux items from state dicts, if found - removes them from unexpected_keys
        """
        import bitsandbytes as bnb

        fp16_statistics_key = param_name.replace("weight", "SCB")
        fp16_weights_format_key = param_name.replace("weight", "weight_format")

        fp16_statistics = state_dict.get(fp16_statistics_key, None)
        fp16_weights_format = state_dict.get(fp16_weights_format_key, None)

        module, tensor_name = get_module_from_name(model, param_name)
        if tensor_name not in module._parameters:
            raise ValueError(f"{module} does not have a parameter or a buffer named {tensor_name}.")

        old_value = getattr(module, tensor_name)

        if not isinstance(module._parameters[tensor_name], bnb.nn.Int8Params):
            raise ValueError(f"Parameter `{tensor_name}` should only be a `bnb.nn.Int8Params` instance.")
        if (
            old_value.device == torch.device("meta")
            and target_device not in ["meta", torch.device("meta")]
            and param_value is None
        ):
            raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {target_device}.")

        new_value = param_value.to("cpu")
        if self.pre_quantized and not self.is_serializable():
            raise ValueError(
                "Detected int8 weights but the version of bitsandbytes is not compatible with int8 serialization. "
                "Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`."
            )

        # Support models using `Conv1D` in place of `nn.Linear` (e.g. openai-community/gpt2) by transposing the weight matrix prior to quantization.
        # Since weights are saved in the correct "orientation", we skip transposing when loading.
        if issubclass(module.source_cls, Conv1D):
            if fp16_statistics is None:
                new_value = new_value.T

        kwargs = old_value.__dict__
        new_value = bnb.nn.Int8Params(new_value, requires_grad=False, **kwargs).to(target_device)

        module._parameters[tensor_name] = new_value
        if fp16_statistics is not None:
            setattr(module.weight, "SCB", fp16_statistics.to(target_device))
            if unexpected_keys is not None:
                unexpected_keys.remove(fp16_statistics_key)

        # We just need to pop the `weight_format` keys from the state dict to remove unneeded
        # messages. The correct format is correctly retrieved during the first forward pass.
        if fp16_weights_format is not None and unexpected_keys is not None:
            unexpected_keys.remove(fp16_weights_format_key)

    def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
        model.is_loaded_in_8bit = True
        model.is_8bit_serializable = self.is_serializable()
        return model

    def _process_model_before_weight_loading(
        self,
        model: "PreTrainedModel",
        device_map,
        keep_in_fp32_modules: List[str] = [],
        **kwargs,
    ):
        from ..integrations import get_keys_to_not_convert, replace_with_bnb_linear

        llm_int8_enable_fp32_cpu_offload = self.quantization_config.llm_int8_enable_fp32_cpu_offload

        # We keep some modules such as the lm_head in their original dtype for numerical stability reasons
        if self.quantization_config.llm_int8_skip_modules is None:
            self.modules_to_not_convert = get_keys_to_not_convert(model)
        else:
            self.modules_to_not_convert = self.quantization_config.llm_int8_skip_modules

        if not isinstance(self.modules_to_not_convert, list):
            self.modules_to_not_convert = [self.modules_to_not_convert]

        self.modules_to_not_convert.extend(keep_in_fp32_modules)

        # Extend `self.modules_to_not_convert` to keys that are supposed to be offloaded to `cpu` or `disk`
        if isinstance(device_map, dict) and len(device_map.keys()) > 1:
            keys_on_cpu = [key for key, value in device_map.items() if value in ["disk", "cpu"]]

            if len(keys_on_cpu) > 0 and not llm_int8_enable_fp32_cpu_offload:
                raise ValueError(
                    "If you want to offload some keys to `cpu` or `disk`, you need to set "
                    "`llm_int8_enable_fp32_cpu_offload=True`. Note that these modules will not be "
                    " converted to 8-bit but kept in 32-bit."
                )
            self.modules_to_not_convert.extend(keys_on_cpu)

        model = replace_with_bnb_linear(
            model, modules_to_not_convert=self.modules_to_not_convert, quantization_config=self.quantization_config
        )
        # TODO: consider bringing replace_with_bnb_linear() code from ..integrations/bitsandbyter.py to here

        model.config.quantization_config = self.quantization_config

    def is_serializable(self, safe_serialization=None):
        _bnb_supports_8bit_serialization = version.parse(importlib.metadata.version("bitsandbytes")) > version.parse(
            "0.37.2"
        )

        if not _bnb_supports_8bit_serialization:
            logger.warning(
                "You are calling `save_pretrained` to a 8-bit converted model, but your `bitsandbytes` version doesn't support it. "
                "If you want to save 8-bit models, make sure to have `bitsandbytes>0.37.2` installed. You will most likely face errors or"
                " unexpected behaviours."
            )
            return False

        return True

    @property
    def is_trainable(self) -> bool:
        return version.parse(importlib.metadata.version("bitsandbytes")) >= version.parse("0.37.0")

    def _dequantize(self, model):
        from ..integrations import dequantize_and_replace

        model = dequantize_and_replace(
            model, self.modules_to_not_convert, quantization_config=self.quantization_config
        )


SUMMARY:
This is the serialized form of a `BertForMaskedLM` checkpoint with 8-bit conversions applied. The main components of the class can be summarized as follows:

*   **Initialization**: The class is used to convert the weights and data for a model from floating-point format to 8-bit (int) format.
*   **Serialization and Deserialization**: The `save_pretrained` method saves the entire model, including weights and configuration. Conversely, the `load_model_from_8bit_checkpoint` method deserializes the pre-trained model and restores it into a normal PyTorch model.
*   **Configuration Management**: A `quantization_config` object is used to define various settings related to quantization, such as whether to use precision 16 or 32 bits for floating-point numbers. There's also an `llm_int8_enable_fp32_cpu_offload` setting that determines whether to disable CPU offloading of some modules during the execution.
*   **Model Processing**: The `save_pretrained` method transforms the model by replacing certain linear layers with custom BNB Linear layers.

Key features of this class include:

1.  Supports 8-bit weight conversion for faster training times and higher performance.
2.  Maintains compatibility across different PyTorch versions (from version v0.37.0 onwards).
3.  Features a `quantization_config` object that defines key settings for optimized quantization.

By utilizing this class, developers can leverage model weights that have been trained efficiently using 8-bit precision to further accelerate their training and processing workflows. Additionally, researchers are able to preserve specific characteristics of complex models that would otherwise become unavailable during the weight conversion process.


key: chroma:document
string_value:
CODE:

class FbgemmFp8HfQuantizer(HfQuantizer):
    """
    FP8 quantization using fbgemm kernels
    """

    requires_parameters_quantization = True
    requires_calibration = False

    required_packages = ["fbgemm-gpu", "accelerate"]

    def __init__(self, quantization_config, **kwargs):
        super().__init__(quantization_config, **kwargs)
        self.quantization_config = quantization_config

    def validate_environment(self, *args, **kwargs):
        if not is_torch_available() or version.parse(importlib.metadata.version("torch")) < version.parse("2.1.0"):
            raise ImportError(
                "Using fbgemm fp8 quantization requires torch > 2.1.0"
                "Please install the latest version of torch ( pip install --upgrade torch )"
            )
        if not is_fbgemm_gpu_available():
            raise ImportError(
                "Using fbgemm fp8 quantization requires fbgemm-gpu library"
                "Please install the latest version of fbgemm-gpu library by following : https://pytorch.org/FBGEMM/fbgemm_gpu-development/InstallationInstructions.html#fbgemm-gpu-install-libraries"
            )

        if not is_accelerate_available("0.32.2"):
            raise ImportError(
                "Loading an FP8 quantized model requires accelerate > 0.32.1 (`pip install --upgrade accelerate`)"
            )

        if not torch.cuda.is_available():
            raise RuntimeError("Using FP8 quantized models with fbgemm kernels requires a GPU")

        compute_capability = torch.cuda.get_device_capability()
        major, minor = compute_capability
        if major < 9:
            raise ValueError(
                "FP8 quantized models is only supported on GPUs with compute capability >= 9.0 (e.g H100)"
            )

        device_map = kwargs.get("device_map", None)
        if device_map is None:
            logger.warning_once(
                "You have loaded an FP8 model on CPU and have a CUDA device available, make sure to set "
                "your model on a GPU device in order to run your model. To remove this warning, pass device_map = 'cuda'. "
            )
        elif device_map is not None:
            if (
                not self.pre_quantized
                and isinstance(device_map, dict)
                and ("cpu" in device_map.values() or "disk" in device_map.values())
            ):
                raise ValueError(
                    "You are attempting to load an FP8 model with a device_map that contains a CPU or disk device."
                    "This is not supported when the model is quantized on the fly. "
                    "Please use a quantized checkpoint or remove the CPU or disk device from the device_map."
                )

    def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
        if torch_dtype is None:
            torch_dtype = torch.bfloat16
            logger.info(
                "Overriding torch_dtype=%s with `torch_dtype=torch.bloat16` due to "
                "requirements of `fbgemm-gpu` to enable model loading in fp8. "
                "Pass your own torch_dtype to specify the dtype of the remaining non-linear layers or pass"
                " torch_dtype=torch.bfloat16 to remove this warning.",
                torch_dtype,
            )
        elif torch_dtype == torch.float16:
            raise ValueError(
                "You cannot use FP8 with torch_dtype=torch.float16."
                "We recommend you passing torch_dtype=torch.bfloat16"
            )
        return torch_dtype

    def check_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        state_dict: Dict[str, Any],
        **kwargs,
    ):
        from ..integrations import FbgemmFp8Linear

        module, tensor_name = get_module_from_name(model, param_name)

        if isinstance(module, FbgemmFp8Linear):
            if self.pre_quantized or tensor_name == "bias":
                if tensor_name == "weight" and param_value.dtype != torch.float8_e4m3fn:
                    raise ValueError("Expect quantized weights but got an unquantized weight")
                return False
            else:
                if tensor_name == "weight_scale":
                    raise ValueError("Expect unquantized weights but got a quantized weight_scale")
                return True
        return False

    def create_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        target_device: "torch.device",
        state_dict: Dict[str, Any],
        unexpected_keys: Optional[List[str]] = None,
    ):
        """
        Quantizes weights into weight and weight_scale
        """
        new_value, weight_scale = torch.ops.fbgemm.quantize_fp8_per_row(param_value)

        module, tensor_name = get_module_from_name(model, param_name)
        module._buffers[tensor_name] = new_value.to(target_device)
        # to have the right output shape -> (out_features, 1)
        module._buffers["weight_scale"] = weight_scale.view(weight_scale.shape[0], 1).to(target_device)

        if unexpected_keys is not None and param_name in unexpected_keys:
            unexpected_keys.remove(param_name)
        del param_name

    def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
        return model

    def _process_model_before_weight_loading(
        self,
        model: "PreTrainedModel",
        device_map,
        keep_in_fp32_modules: List[str] = [],
        **kwargs,
    ):
        from ..integrations import get_keys_to_not_convert, replace_with_fbgemm_fp8_linear

        self.modules_to_not_convert = get_keys_to_not_convert(model)

        if self.quantization_config.modules_to_not_convert is not None:
            self.modules_to_not_convert.extend(self.quantization_config.modules_to_not_convert)

        model = replace_with_fbgemm_fp8_linear(
            model,
            modules_to_not_convert=self.modules_to_not_convert,
            quantization_config=self.quantization_config,
            pre_quantized=self.pre_quantized,
        )

        model.config.quantization_config = self.quantization_config

    def update_missing_keys(self, model, missing_keys: List[str], prefix: str) -> List[str]:
        from ..integrations import FbgemmFp8Linear

        not_missing_keys = []
        for name, module in model.named_modules():
            if isinstance(module, FbgemmFp8Linear):
                for missing in missing_keys:
                    if (
                        (name in missing or name in f"{prefix}.{missing}")
                        and not missing.endswith(".weight")
                        and not missing.endswith(".bias")
                    ):
                        not_missing_keys.append(missing)
        return [k for k in missing_keys if k not in not_missing_keys]

    def is_serializable(self, safe_serialization=None):
        return True

    @property
    def is_trainable(self) -> bool:


SUMMARY:
**Class Summary: FbgemmFp8HfQuantizer**

**Purpose and Functionality**
---------------------------

The `FbgemmFp8HfQuantizer` class is a quantization module for PyTorch models using the FBGEMM kernel. It provides functionality for FP8 quantization, calibration, and verification.

**Key Implementation Details**
-------------------------------

* Validates environment setup, including Torch and FBGEMM-GPU availability.
* Updates `torch_dtype` to `torch.bfloat16` due to FBGEMM hardware requirement.
* Checks if a model's parameters are already quantized or not.
* Quantizes weights into weight and weight_scale.

**Notable Dependencies and Requirements**
-----------------------------------------

* PyTorch >= 2.1.0
* FBGEMM-GPU library
* Accelerate library (>= 0.32.2)
*CUDA GPU with compute capability >= 9.0

**Class Methods and Variables**
------------------------------

* `__init__`: Initializes the quantizer with a quantization config.
* `_process_model_before_weight_loading` and `_process_model_after_weight_loading`: Processing methods for model initialization.

**Constraints and Inheritance**
-----------------------------

* Not trainable.
* Serializable by default ( overridden method).
* Requires calibration: False.


key: chroma:document
string_value:
CODE:

class DataCollatorMixin:
    def __call__(self, features, return_tensors=None):
        if return_tensors is None:
            return_tensors = self.return_tensors
        if return_tensors == "tf":
            return self.tf_call(features)
        elif return_tensors == "pt":
            return self.torch_call(features)
        elif return_tensors == "np":
            return self.numpy_call(features)
        else:


SUMMARY:
**Summary: DataCollatorMixin Class**

The `DataCollatorMixin` class is designed to be mixed into datasets for efficient data augmentation. Its primary purpose is to collate and preprocess dataset features into tensors compatible with specific deep learning frameworks (PyTorch, TensorFlow).

**Main Purpose and Functionality**
- Primary function: Collates and preprocesses dataset features for different deep learning frameworks.
- Takes in `features` and optional `return_tensors` parameter.

**Key Implementation Details**

* Utilizes the `self.return_tensors` attribute to determine the output format internally if no value is provided for `return_tensors`.
* Supports conversions to PyTorch, TensorFlow, or NumPy formats.

**Notable Dependencies or Requirements**

- Relies on framework-specific interfaces:
  - `torch_call(features)`
  - `tf_call(features)`
  - `numpy_call(features)`
- Assuming existence of these methods in the class implementing this mixin.

**Interfaces and Inherited Behavior**
- Implies implementation of a generic dataset interface.
- Assumes interaction with frameworks like PyTorch (`pt`), TensorFlow (`tf`), or NumPy (`np`).


key: chroma:document
string_value:
CODE:

def pad_without_fast_tokenizer_warning(tokenizer, *pad_args, **pad_kwargs):
    """
    Pads without triggering the warning about how using the pad function is sub-optimal when using a fast tokenizer.
    """

    # To avoid errors when using Feature extractors
    if not hasattr(tokenizer, "deprecation_warnings"):
        return tokenizer.pad(*pad_args, **pad_kwargs)

    # Save the state of the warning, then disable it
    warning_state = tokenizer.deprecation_warnings.get("Asking-to-pad-a-fast-tokenizer", False)
    tokenizer.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = True

    try:
        padded = tokenizer.pad(*pad_args, **pad_kwargs)
    finally:
        # Restore the state of the warning.
        tokenizer.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = warning_state



SUMMARY:
**Function Summary: Pad Without Fast Tokenizer Warning**

### Purpose and Functionality
This function, `pad_without_fast_tokenizer_warning`, pads input text using a tokenizer while avoiding the deprecation warning for fast tokenizers.

### Key Implementation Details
- Uses the `pad` method of a tokenizer to pad input text.
- Saves/ restoring the state of a deprecation warning.

### Notable Dependencies or Requirements
- `tokenizer` object with a `deprecation_warnings` attribute and a `pad` method.


key: chroma:document
string_value:
CODE:

def default_data_collator(features: List[InputDataClass], return_tensors="pt") -> Dict[str, Any]:
    """
    Very simple data collator that simply collates batches of dict-like objects and performs special handling for
    potential keys named:

        - `label`: handles a single value (int or float) per object
        - `label_ids`: handles a list of values per object

    Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs
    to the model. See glue and ner for example of how it's useful.
    """

    # In this function we'll make the assumption that all `features` in the batch
    # have the same attributes.
    # So we will look at the first element as a proxy for what attributes exist
    # on the whole batch.

    if return_tensors == "pt":
        return torch_default_data_collator(features)
    elif return_tensors == "tf":
        return tf_default_data_collator(features)
    elif return_tensors == "np":


SUMMARY:
**Summary**

The `default_data_collator` function collates batches of dictionaries, handling special cases for `label` and `label_ids` keys. It performs no additional preprocessing.

**Main Purpose and Functionality**
```python
def default_data_collator(features: List[InputDataClass], return_tensors="pt") -> Dict[str, Any]:
```
Converts batch of input dictionaries into a collated data structure, suitable for model input.

**Key Implementation Details**

* Handles `label` key with single value (int or float)
* Handles `label_ids` key with list of values per object
* Performs no additional preprocessing

**Notable Dependencies or Requirements**

* Returns tensors in one of the following formats:
	+ `pt` (PyTorch tensor)
	+ `tf` (TensorFlow tensor)
	+ `np` (NumPy array)


key: chroma:document
string_value:
CODE:

def torch_default_data_collator(features: List[InputDataClass]) -> Dict[str, Any]:
    import torch

    if not isinstance(features[0], Mapping):
        features = [vars(f) for f in features]
    first = features[0]
    batch = {}

    # Special handling for labels.
    # Ensure that tensor is created with the correct type
    # (it should be automatically the case, but let's make sure of it.)
    if "label" in first and first["label"] is not None:
        label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
        dtype = torch.long if isinstance(label, int) else torch.float
        batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
    elif "label_ids" in first and first["label_ids"] is not None:
        if isinstance(first["label_ids"], torch.Tensor):
            batch["labels"] = torch.stack([f["label_ids"] for f in features])
        else:
            dtype = torch.long if isinstance(first["label_ids"][0], int) else torch.float
            batch["labels"] = torch.tensor([f["label_ids"] for f in features], dtype=dtype)

    # Handling of all other possible keys.
    # Again, we will use the first element to figure out which key/values are not None for this model.
    for k, v in first.items():
        if k not in ("label", "label_ids") and v is not None and not isinstance(v, str):
            if isinstance(v, torch.Tensor):
                batch[k] = torch.stack([f[k] for f in features])
            elif isinstance(v, np.ndarray):
                batch[k] = torch.from_numpy(np.stack([f[k] for f in features]))
            else:
                batch[k] = torch.tensor([f[k] for f in features])



SUMMARY:
**torch_default_data_collator Function Summary**

### Main Purpose and Functionality

Defines a data collator function for PyTorch, which processes input data into a dictionary format suitable for batching. The primary purpose is to extract relevant features from the input `features` list and transform them into tensors while handling various data types (e.g., labels, images).

### Key Implementation Details

1. **Special Handling for Labels**:
	* Automatically creates tensor with correct type using `torch.tensor`.
	* Uses `first["label"]` to determine label type (int or float) and batch labels accordingly.
2. **Handling of Other Feature Values**:
	* Iterates through the first element's key-value pairs to identify non-None values that are not strings.
	* Utilizes `torch.stack` or `torch.tensor` to convert these values into tensors.

### Notable Dependencies and Requirements

* PyTorch
* NumPy (imported as `np`)
* InputDataClass (class used in the function, exact name not specified)

### Method Signatures and Interfaces

* `features: List[InputDataClass] -> Dict[str, Any]`: Inputs a list of `InputDataClass` objects.
* `batch: Dict[str, torch.Tensor]`: Outputs a dictionary with feature values as tensors.

### External Dependencies (inferred)

* None explicitly mentioned, but potentially relies on custom `InputDataClass` definition.


key: chroma:document
string_value:
CODE:

def tf_default_data_collator(features: List[InputDataClass]) -> Dict[str, Any]:
    import tensorflow as tf

    if not isinstance(features[0], Mapping):
        features = [vars(f) for f in features]
    first = features[0]
    batch = {}

    # Special handling for labels.
    # Ensure that tensor is created with the correct type
    # (it should be automatically the case, but let's make sure of it.)
    if "label" in first and first["label"] is not None:
        label_col_name = "label"
    elif "label_ids" in first and first["label_ids"] is not None:
        label_col_name = "label_ids"
    elif "labels" in first and first["labels"] is not None:
        label_col_name = "labels"
    else:
        label_col_name = None
    if label_col_name is not None:
        if isinstance(first[label_col_name], tf.Tensor):
            dtype = tf.int64 if first[label_col_name].dtype.is_integer else tf.float32
        elif isinstance(first[label_col_name], np.ndarray) or isinstance(first[label_col_name], np.generic):
            dtype = tf.int64 if np.issubdtype(first[label_col_name].dtype, np.integer) else tf.float32
        elif isinstance(first[label_col_name], (tuple, list)):
            dtype = tf.int64 if isinstance(first[label_col_name][0], int) else tf.float32
        else:
            dtype = tf.int64 if isinstance(first[label_col_name], int) else tf.float32
        batch["labels"] = tf.convert_to_tensor([f[label_col_name] for f in features], dtype=dtype)
    # Handling of all other possible keys.
    # Again, we will use the first element to figure out which key/values are not None for this model.
    for k, v in first.items():
        if k not in ("label", "label_ids", "labels") and v is not None and not isinstance(v, str):
            if isinstance(v, (tf.Tensor, np.ndarray)):
                batch[k] = tf.stack([f[k] for f in features])
            else:
                batch[k] = tf.convert_to_tensor([f[k] for f in features])



SUMMARY:
**Function Summary: `tf_default_data_collator`**

**Purpose and Functionality:**
The `tf_default_data_collator` function collates input data (`features`) into batches, handling variables such as labels, tensor fields, and other values.

**Main Purpose:** Create a batch of data from a list of input features.

**Key Implementation Details:**

* Checks for label types (tensor, numpy array or generic) and converts to integer64 or float32 dtype.
* Handles various key-value pairs not covered by labels.
* Uses stack() or convert_to_tensor() to create batch values from raw inputs.

**Notable Dependencies and Requirements:**

* TensorFlow library (`tf`)
* NumPy library (__imported in `np.issubdtype` usage)
 
**Interface Hierarchy:** 
None


key: chroma:document
string_value:
CODE:

def numpy_default_data_collator(features: List[InputDataClass]) -> Dict[str, Any]:
    if not isinstance(features[0], Mapping):
        features = [vars(f) for f in features]
    first = features[0]
    batch = {}

    # Special handling for labels.
    # Ensure that tensor is created with the correct type
    # (it should be automatically the case, but let's make sure of it.)
    if "label" in first and first["label"] is not None:
        label = first["label"].item() if isinstance(first["label"], np.ndarray) else first["label"]
        dtype = np.int64 if isinstance(label, int) else np.float32
        batch["labels"] = np.array([f["label"] for f in features], dtype=dtype)
    elif "label_ids" in first and first["label_ids"] is not None:
        if isinstance(first["label_ids"], np.ndarray):
            batch["labels"] = np.stack([f["label_ids"] for f in features])
        else:
            dtype = np.int64 if isinstance(first["label_ids"][0], int) else np.float32
            batch["labels"] = np.array([f["label_ids"] for f in features], dtype=dtype)

    # Handling of all other possible keys.
    # Again, we will use the first element to figure out which key/values are not None for this model.
    for k, v in first.items():
        if k not in ("label", "label_ids") and v is not None and not isinstance(v, str):
            if isinstance(v, np.ndarray):
                batch[k] = np.stack([f[k] for f in features])
            else:
                batch[k] = np.array([f[k] for f in features])



SUMMARY:
**Function Summary:**

`numpy_default_data_collator`

Creates a data collation dictionary (`batch`) from input `features`, handling labels and other key-value pairs.

**Primary Purpose and Functionality:**

Collates dictionary values into NumPy arrays based on the first feature's keys, ensuring correct types for labels.

**Key Implementation Details:**

- Handles special cases for "label" and "label_ids" in features
- Uses NumPy arrays to stack values of key-value pairs
- Preserves original data types (int64 or float32) for numerical values

**Notable Dependencies/Requirements:**

- `numpy` library for array creation and manipulation
- `InputDataClass` class for type inference
- `Dict[str, Any]` type hint for return value


key: chroma:document
string_value:
CODE:

def _torch_collate_batch(examples, tokenizer, pad_to_multiple_of: Optional[int] = None):
    """Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary."""
    import torch

    # Tensorize if necessary.
    if isinstance(examples[0], (list, tuple, np.ndarray)):
        examples = [torch.tensor(e, dtype=torch.long) for e in examples]

    length_of_first = examples[0].size(0)

    # Check if padding is necessary.

    are_tensors_same_length = all(x.size(0) == length_of_first for x in examples)
    if are_tensors_same_length and (pad_to_multiple_of is None or length_of_first % pad_to_multiple_of == 0):
        if not isinstance(examples, torch.Tensor):
            return torch.stack(examples, dim=0)

    # If yes, check if we have a `pad_token`.
    if tokenizer.pad_token is None:
        raise ValueError(
            "You are attempting to pad samples but the tokenizer you are using"
            f" ({tokenizer.__class__.__name__}) does not have a pad token."
        )

    # Creating the full tensor and filling it with our data.
    max_length = max(x.size(0) for x in examples)
    if pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
        max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
    result = examples[0].new_full([len(examples), max_length], tokenizer.pad_token_id)
    for i, example in enumerate(examples):
        if tokenizer.padding_side == "right":
            result[i, : example.shape[0]] = example
        else:
            result[i, -example.shape[0] :] = example


SUMMARY:
**`_torch_collate_batch` Function Summary**

* **Purpose:** Collates input data `examples` into a batch using tokenizer information for padding and tensorization.

### Implementation Details

* Uses PyTorch tensors to maintain consistency.
* Tensorizes input data if necessary (list, tuple, or NumPy array).
* Applies padding based on the provided `pad_to_multiple_of` parameter.

**Method Signatures:**

* `(examples, tokenizer, pad_to_multiple_of) -> torch.Tensor`

### Dependencies and Requirements

* PyTorch
* Tokenizer with a defined `pad_token`
* `Optional[int]` from `typing` for `pad_to_multiple_of` parameter

### Key Design Choices

* Uses `torch.tensor` to convert data into tensors.
* Applies padding using the tokenizer's `padding_side` attribute.


key: chroma:document
string_value:
CODE:

def _tf_collate_batch(examples, tokenizer, pad_to_multiple_of: Optional[int] = None):
    import tensorflow as tf

    """Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary."""
    # Tensorize if necessary.
    if isinstance(examples[0], (list, tuple)):
        examples = [tf.convert_to_tensor(e, dtype=tf.int64) for e in examples]

    # Check if padding is necessary.
    length_of_first = len(examples[0])
    are_tensors_same_length = all(len(x) == length_of_first for x in examples)
    if are_tensors_same_length and (pad_to_multiple_of is None or length_of_first % pad_to_multiple_of == 0):
        return tf.stack(examples, axis=0)

    # If yes, check if we have a `pad_token`.
    if tokenizer.pad_token is None:
        raise ValueError(
            "You are attempting to pad samples but the tokenizer you are using"
            f" ({tokenizer.__class__.__name__}) does not have a pad token."
        )

    # Creating the full tensor and filling it with our data.
    max_length = max(len(x) for x in examples)
    if pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
        max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
    # result = examples[0].new_full([len(examples), max_length], tokenizer.pad_token_id)
    result = []
    rank = tf.rank(examples[0])
    paddings = np.zeros((rank, 2), dtype=np.int32)
    for example in examples:
        if tokenizer.padding_side == "right":
            paddings[0, 1] = max_length - len(example)
        else:
            paddings[0, 0] = max_length - len(example)
        result.append(tf.pad(example, paddings, constant_values=tokenizer.pad_token_id))


SUMMARY:
**Function Summary: `_tf_collate_batch`**

* **Purpose:** Collates Python `examples` into a batch, utilizing the provided `tokenizer` for padding if necessary.

**Key Implementation Details:**
- Utilizes TensorFlow for tensorization and stacking.
- Checks if padding is necessary based on tokenizer's `pad_token`.
- Dynamically adjusts maximum length to accommodate padding requirement.
- Employs left/right padding depending on tokenizer's `padding_side`.

**Method Signatures:**
- `_tf_collate_batch(examples, tokenizer, pad_to_multiple_of=Optional[int] = None)`

**Notable Dependencies/Requirements:**
- `tensorflow` (TF)
- `tokenizer` object with `pad_token`, `padding_side`, and `new_full` methods.
- External dependency on NumPy (`np`) for padding creation.


key: chroma:document
string_value:
CODE:

def _numpy_collate_batch(examples, tokenizer, pad_to_multiple_of: Optional[int] = None):
    """Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary."""
    # Tensorize if necessary.
    if isinstance(examples[0], (list, tuple)):
        examples = [np.array(e, dtype=np.int64) for e in examples]

    # Check if padding is necessary.
    length_of_first = len(examples[0])
    are_tensors_same_length = all(len(x) == length_of_first for x in examples)
    if are_tensors_same_length and (pad_to_multiple_of is None or length_of_first % pad_to_multiple_of == 0):
        return np.stack(examples, axis=0)

    # If yes, check if we have a `pad_token`.
    if tokenizer.pad_token is None:
        raise ValueError(
            "You are attempting to pad samples but the tokenizer you are using"
            f" ({tokenizer.__class__.__name__}) does not have a pad token."
        )

    # Creating the full tensor and filling it with our data.
    max_length = max(len(x) for x in examples)
    if pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
        max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
    result = np.full(shape=(len(examples), max_length), fill_value=tokenizer.pad_token_id, dtype=examples[0].dtype)
    for i, example in enumerate(examples):
        if tokenizer.padding_side == "right":
            result[i, : example.shape[0]] = example
        else:
            result[i, -example.shape[0] :] = example


SUMMARY:
**Summary:**

The `_numpy_collate_batch` function collates a list of examples into a batch. It takes `examples`, `tokenizer`, and optional parameters to pad the tensor.

**Main Purpose and Functionality:**
Collates input data into a padded batch using the provided tokenizer for adjustment.

**Key Implementation Details:**

*   Tensors are converted to NumPy arrays if necessary.
*   The function checks if padding is required based on the length of tensors in the list.
*   If no padding is needed, it stacks the tensors along a zero axis (axis 0).
*   Otherwise, it creates a full tensor with the pad token and fills it with data from the example samples.

**Notable Method Signatures and Interfaces:**

*   `np.array(e, dtype=np.int64)`: converts lists or tuples to NumPy arrays.
*   `np.stack(examples, axis=0)`: stacks tensors along an axis (in this case axis 0).
*   `tokenizer.pad_token`: retrieves the tokenizer's pad token.

**Notable Dependencies and Requirements:**

*   NumPy (`np`): used for array conversions and stacking.
*   Tokenizer class: expected to have methods like `pad_token`, `padding_side`, etc. depending on its implementation.

**External Dependencies:**

*   The input data is assumed to be in a format compatible with the tokenizer, but no specific dependencies are mentioned in this code snippet.


key: chroma:document
string_value:
CODE:

def tolist(x):
    if isinstance(x, list):
        return x
    elif hasattr(x, "numpy"):  # Checks for TF tensors without needing the import
        x = x.numpy()


SUMMARY:
**Summary:**

The `tolist` function converts an input object to its equivalent list representation. It checks if the input is already a list, in which case it returns it directly. If not, it checks if the object has a `numpy` attribute, suggesting it's a TensorFlow tensor. In this case, it uses the `numpy` method to retrieve the underlying numpy array and then converts it back to a list using the `tolist` method.

**Key Implementation Details:**

* Checks for list literal or object with `numpy` attribute
* Utilizes `numpy` method on tensors to retrieve underlying numpy array

**Notable Dependencies/Requirements:**

* `numpy` (TF tensor specific implementation)
* List literals or objects with `numpy` attribute


key: chroma:document
string_value:
CODE:

class Split(Enum):
    train = "train"


SUMMARY:
**Class Summary:**

`Split(Enum)` - Enumerates possible split options for data.

*   **Purpose:** Provides a way to distinguish between training and development datasets, allowing for more efficient splitting of data during model training.

    **Implementation Details:** 
    Used `Enum` class to create an enumeration with two possible values: `"train"` and `"dev"`.
    
   


key: chroma:document
string_value:
CODE:

class SquadDataset(Dataset):
    """
    This will be superseded by a framework-agnostic approach soon.
    """

    args: SquadDataTrainingArguments
    features: List[SquadFeatures]
    mode: Split
    is_language_sensitive: bool

    def __init__(
        self,
        args: SquadDataTrainingArguments,
        tokenizer: PreTrainedTokenizer,
        limit_length: Optional[int] = None,
        mode: Union[str, Split] = Split.train,
        is_language_sensitive: Optional[bool] = False,
        cache_dir: Optional[str] = None,
        dataset_format: Optional[str] = "pt",
    ):
        self.args = args
        self.is_language_sensitive = is_language_sensitive
        self.processor = SquadV2Processor() if args.version_2_with_negative else SquadV1Processor()
        if isinstance(mode, str):
            try:
                mode = Split[mode]
            except KeyError:
                raise KeyError("mode is not a valid split name")
        self.mode = mode
        # Load data features from cache or dataset file
        version_tag = "v2" if args.version_2_with_negative else "v1"
        cached_features_file = os.path.join(
            cache_dir if cache_dir is not None else args.data_dir,
            f"cached_{mode.value}_{tokenizer.__class__.__name__}_{args.max_seq_length}_{version_tag}",
        )

        # Make sure only the first process in distributed training processes the dataset,
        # and the others will use the cache.
        lock_path = cached_features_file + ".lock"
        with FileLock(lock_path):
            if os.path.exists(cached_features_file) and not args.overwrite_cache:
                start = time.time()
                self.old_features = torch.load(cached_features_file)

                # Legacy cache files have only features, while new cache files
                # will have dataset and examples also.
                self.features = self.old_features["features"]
                self.dataset = self.old_features.get("dataset", None)
                self.examples = self.old_features.get("examples", None)
                logger.info(
                    f"Loading features from cached file {cached_features_file} [took %.3f s]", time.time() - start
                )

                if self.dataset is None or self.examples is None:
                    logger.warning(
                        f"Deleting cached file {cached_features_file} will allow dataset and examples to be cached in"
                        " future run"
                    )
            else:
                if mode == Split.dev:
                    self.examples = self.processor.get_dev_examples(args.data_dir)
                else:
                    self.examples = self.processor.get_train_examples(args.data_dir)

                self.features, self.dataset = squad_convert_examples_to_features(
                    examples=self.examples,
                    tokenizer=tokenizer,
                    max_seq_length=args.max_seq_length,
                    doc_stride=args.doc_stride,
                    max_query_length=args.max_query_length,
                    is_training=mode == Split.train,
                    threads=args.threads,
                    return_dataset=dataset_format,
                )

                start = time.time()
                torch.save(
                    {"features": self.features, "dataset": self.dataset, "examples": self.examples},
                    cached_features_file,
                )
                # ^ This seems to take a lot of time so I want to investigate why and how we can improve.
                logger.info(
                    f"Saving features into cached file {cached_features_file} [took {time.time() - start:.3f} s]"
                )

    def __len__(self):
        return len(self.features)

    def __getitem__(self, i) -> Dict[str, torch.Tensor]:
        # Convert to Tensors and build dataset
        feature = self.features[i]

        input_ids = torch.tensor(feature.input_ids, dtype=torch.long)
        attention_mask = torch.tensor(feature.attention_mask, dtype=torch.long)
        token_type_ids = torch.tensor(feature.token_type_ids, dtype=torch.long)
        cls_index = torch.tensor(feature.cls_index, dtype=torch.long)
        p_mask = torch.tensor(feature.p_mask, dtype=torch.float)
        is_impossible = torch.tensor(feature.is_impossible, dtype=torch.float)

        inputs = {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
            "token_type_ids": token_type_ids,
        }

        if self.args.model_type in ["xlm", "roberta", "distilbert", "camembert"]:
            del inputs["token_type_ids"]

        if self.args.model_type in ["xlnet", "xlm"]:
            inputs.update({"cls_index": cls_index, "p_mask": p_mask})
            if self.args.version_2_with_negative:
                inputs.update({"is_impossible": is_impossible})
            if self.is_language_sensitive:
                inputs.update({"langs": (torch.ones(input_ids.shape, dtype=torch.int64) * self.args.lang_id)})

        if self.mode == Split.train:
            start_positions = torch.tensor(feature.start_position, dtype=torch.long)
            end_positions = torch.tensor(feature.end_position, dtype=torch.long)
            inputs.update({"start_positions": start_positions, "end_positions": end_positions})



SUMMARY:
**SquadDataset Class Summary**

The `SquadDataset` class is a PyTorch dataset implementation for the SQuAD (Stanford Question Answering Dataset) pipeline. It inherits from `Dataset`.

**Primary Purpose:**
Load, preprocess, and cache SQuAD data features for use in question answering models.

**Key Features and Functions:**

* Initializes the dataset with arguments, tokenizer, and cache directory
* Loads existing cached features or generates new ones based on input data and configuration
* Provides methods to access the dataset elements (features, examples, etc.)

**Implementation Details:**

* Uses `SquadV2Processor` or `SquadV1Processor` for data preprocessing
* Supports language-sensitive feature generation
* Caches processed features using a file lock mechanism

**Notable Dependencies and Interfaces:**

* `PreTrainedTokenizer` from Hugging Face Transformers library
* `SquadDataTrainingArguments` for dataset configuration
* `Split` enum for dataset split (dev, train)

**External Dependencies:**

* `torch` and `logger` from PyTorch library
* `os`, `time`, and `FileLock` from Python standard library


key: chroma:document
string_value:
CODE:

class TextDataset(Dataset):
    """
    This will be superseded by a framework-agnostic approach soon.
    """

    def __init__(
        self,
        tokenizer: PreTrainedTokenizer,
        file_path: str,
        block_size: int,
        overwrite_cache=False,
        cache_dir: Optional[str] = None,
    ):
        warnings.warn(
            DEPRECATION_WARNING.format(
                "https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py"
            ),
            FutureWarning,
        )
        if os.path.isfile(file_path) is False:
            raise ValueError(f"Input file path {file_path} not found")

        block_size = block_size - tokenizer.num_special_tokens_to_add(pair=False)

        directory, filename = os.path.split(file_path)
        cached_features_file = os.path.join(
            cache_dir if cache_dir is not None else directory,
            f"cached_lm_{tokenizer.__class__.__name__}_{block_size}_{filename}",
        )

        # Make sure only the first process in distributed training processes the dataset,
        # and the others will use the cache.
        lock_path = cached_features_file + ".lock"
        with FileLock(lock_path):
            if os.path.exists(cached_features_file) and not overwrite_cache:
                start = time.time()
                with open(cached_features_file, "rb") as handle:
                    self.examples = pickle.load(handle)
                logger.info(
                    f"Loading features from cached file {cached_features_file} [took %.3f s]", time.time() - start
                )

            else:
                logger.info(f"Creating features from dataset file at {directory}")

                self.examples = []
                with open(file_path, encoding="utf-8") as f:
                    text = f.read()

                tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))

                for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
                    self.examples.append(
                        tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size])
                    )
                # Note that we are losing the last truncated example here for the sake of simplicity (no padding)
                # If your dataset is small, first you should look for a bigger one :-) and second you
                # can change this behavior by adding (model specific) padding.

                start = time.time()
                with open(cached_features_file, "wb") as handle:
                    pickle.dump(self.examples, handle, protocol=pickle.HIGHEST_PROTOCOL)
                logger.info(
                    f"Saving features into cached file {cached_features_file} [took {time.time() - start:.3f} s]"
                )

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> torch.Tensor:


SUMMARY:
**Summary**

The `TextDataset` class is a PyTorch dataset implementation for natural language processing tasks. It loads text data from a file and tokenizes it using a pre-trained tokenizer.

**Key Implementation Details**

* Loads text data from a file and tokenizes it using a pre-trained tokenizer (`PreTrainedTokenizer`)
* Truncates the tokenized text in blocks of `block_size` to manage memory
* Cache features to disk for faster loading on subsequent runs

**Notable Dependencies/Requirements**

* `transformers`: Hugging Face Transformers library (specifically, `PreTrainedTokenizer`)

**Interfaces and Method Signatures**

* `__init__(tokenizer: PreTrainedTokenizer, file_path: str, block_size: int)`: Initializes the dataset with tokenizer, file path, and block size
* `__len__()`: Returns the number of examples in the dataset
* `__getitem__(i) -> torch.Tensor`: Returns a tensor representation of the text data at index `i`


key: chroma:document
string_value:
CODE:

class LineByLineTextDataset(Dataset):
    """
    This will be superseded by a framework-agnostic approach soon.
    """

    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
        warnings.warn(
            DEPRECATION_WARNING.format(
                "https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py"
            ),
            FutureWarning,
        )
        if os.path.isfile(file_path) is False:
            raise ValueError(f"Input file path {file_path} not found")
        # Here, we do not cache the features, operating under the assumption
        # that we will soon use fast multithreaded tokenizers from the
        # `tokenizers` repo everywhere =)
        logger.info(f"Creating features from dataset file at {file_path}")

        with open(file_path, encoding="utf-8") as f:
            lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]

        batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)
        self.examples = batch_encoding["input_ids"]
        self.examples = [{"input_ids": torch.tensor(e, dtype=torch.long)} for e in self.examples]

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> Dict[str, torch.tensor]:


SUMMARY:
**Summary:**

* **Class:** `LineByLineTextDataset`
* **Purpose:** A PyTorch dataset class for loading and processing text data line-by-line.

**Main Functionality:**

The class loads a text file, tokenizes it using the provided `tokenizer`, and returns a batch of encoded examples. The encoded batch is returned as a list of dictionaries containing `input_ids`.

**Implementation Details:**

* The tokenizer is used to split the text into subsequences (batch_encoding) with special tokens (`add_special_tokens=True`) and truncation at max length `block_size`.
* The class caches no features.
* The dataset returns examples in the format `{"input_ids": torch.tensor(e, dtype=torch.long)}`.

**Notable Dependencies:**

* `PreTrainedTokenizer`
* `os.path.isfile`

**Interfaces Implemented:**

* PyTorch's `Dataset` interface

**External Requirements: None mentioned**


key: chroma:document
string_value:
CODE:

class LineByLineWithRefDataset(Dataset):
    """
    This will be superseded by a framework-agnostic approach soon.
    """

    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int, ref_path: str):
        warnings.warn(
            DEPRECATION_WARNING.format(
                "https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm_wwm.py"
            ),
            FutureWarning,
        )
        if os.path.isfile(file_path) is False:
            raise ValueError(f"Input file path {file_path} not found")
        if os.path.isfile(ref_path) is False:
            raise ValueError(f"Ref file path {file_path} not found")
        # Here, we do not cache the features, operating under the assumption
        # that we will soon use fast multithreaded tokenizers from the
        # `tokenizers` repo everywhere =)
        logger.info(f"Creating features from dataset file at {file_path}")
        logger.info(f"Use ref segment results at {ref_path}")
        with open(file_path, encoding="utf-8") as f:
            data = f.readlines()  # use this method to avoid delimiter '\u2029' to split a line
        data = [line.strip() for line in data if len(line) > 0 and not line.isspace()]
        # Get ref inf from file
        with open(ref_path, encoding="utf-8") as f:
            ref = [json.loads(line) for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
        if len(data) != len(ref):
            raise ValueError(
                f"Length of Input file should be equal to Ref file. But the length of {file_path} is {len(data)} "
                f"while length of {ref_path} is {len(ref)}"
            )

        batch_encoding = tokenizer(data, add_special_tokens=True, truncation=True, max_length=block_size)
        self.examples = batch_encoding["input_ids"]
        self.examples = [{"input_ids": torch.tensor(e, dtype=torch.long)} for e in self.examples]

        n = len(self.examples)
        for i in range(n):
            self.examples[i]["chinese_ref"] = torch.tensor(ref[i], dtype=torch.long)

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> Dict[str, torch.tensor]:


SUMMARY:
**Class Summary: LineByLineWithRefDataset**

**Main Purpose and Functionality**
=====================================

The `LineByLineWithRefDataset` class is a dataset class for storing and processing text data, specifically designed to work with a tokenization framework. It reads input data from a file, maps it to reference data stored in another file, and returns a batched tensor representation of the data.

**Key Implementation Details**
================================

* The `__init__` method initializes the dataset by reading input data from a file and reference data from another file.
* It uses an encoder (`PreTrainedTokenizer`) to tokenize the input data and process it into a format suitable for input IDs`.
* The `self.examples` attribute stores the processed input IDs, which are stored as tensors of type torch.long`.
* The `self.examples[i]["chinese_ref"]` attribute stores the corresponding reference values as tensors.

**Notable Dependencies or Requirements**
============================================

* `PreTrainedTokenizer`: used for tokenization
* External files: `input_file_path` and `ref_file_path`
* Hugging Face transformer framework for tokenization (deprecation warning)

**Method Summaries**

* `__len__`: returns the length of the dataset.
* `__getitem__`: returns a single example from the dataset as a dictionary with input IDs and reference values.

### Context Example
```python
dataset = LineByLineWithRefDataset(tokenizer, 'input_data.txt', block_size, 'reference_data.json')
example = dataset[0]
print(example)  # outputs: {'input_ids': tensor of type torch.long, 'chinese_ref': tensor of type torch.long}
```
This code demonstrates how to create an instance of the `LineByLineWithRefDataset` class and retrieve a single example from it.


key: chroma:document
string_value:
CODE:

class LineByLineWithSOPTextDataset(Dataset):
    """
    Dataset for sentence order prediction task, prepare sentence pairs for SOP task
    """

    def __init__(self, tokenizer: PreTrainedTokenizer, file_dir: str, block_size: int):
        warnings.warn(
            DEPRECATION_WARNING.format(
                "https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py"
            ),
            FutureWarning,
        )
        if os.path.isdir(file_dir) is False:
            raise ValueError(f"{file_dir} is not a directory")
        logger.info(f"Creating features from dataset file folder at {file_dir}")
        self.examples = []
        # TODO: randomness could apply a random seed, ex. rng = random.Random(random_seed)
        # file path looks like ./dataset/wiki_1, ./dataset/wiki_2
        for file_name in os.listdir(file_dir):
            file_path = os.path.join(file_dir, file_name)
            if os.path.isfile(file_path) is False:
                raise ValueError(f"{file_path} is not a file")
            article_open = False
            with open(file_path, encoding="utf-8") as f:
                original_lines = f.readlines()
                article_lines = []
                for line in original_lines:
                    if "<doc id=" in line:
                        article_open = True
                    elif "</doc>" in line:
                        article_open = False
                        document = [
                            tokenizer.convert_tokens_to_ids(tokenizer.tokenize(line))
                            for line in article_lines[1:]
                            if (len(line) > 0 and not line.isspace())
                        ]

                        examples = self.create_examples_from_document(document, block_size, tokenizer)
                        self.examples.extend(examples)
                        article_lines = []
                    else:
                        if article_open:
                            article_lines.append(line)

        logger.info("Dataset parse finished.")

    def create_examples_from_document(self, document, block_size, tokenizer, short_seq_prob=0.1):
        """Creates examples for a single document."""

        # Account for special tokens
        max_num_tokens = block_size - tokenizer.num_special_tokens_to_add(pair=True)

        # We *usually* want to fill up the entire sequence since we are padding
        # to `block_size` anyways, so short sequences are generally wasted
        # computation. However, we *sometimes*
        # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter
        # sequences to minimize the mismatch between pretraining and fine-tuning.
        # The `target_seq_length` is just a rough target however, whereas
        # `block_size` is a hard limit.
        target_seq_length = max_num_tokens
        if random.random() < short_seq_prob:
            target_seq_length = random.randint(2, max_num_tokens)

        # We DON'T just concatenate all of the tokens from a document into a long
        # sequence and choose an arbitrary split point because this would make the
        # next sentence prediction task too easy. Instead, we split the input into
        # segments "A" and "B" based on the actual "sentences" provided by the user
        # input.
        examples = []
        current_chunk = []  # a buffer stored current working segments
        current_length = 0
        i = 0
        while i < len(document):
            segment = document[i]  # get a segment
            if not segment:
                i += 1
                continue
            current_chunk.append(segment)  # add a segment to current chunk
            current_length += len(segment)  # overall token length
            # if current length goes to the target length or reaches the end of file, start building token a and b
            if i == len(document) - 1 or current_length >= target_seq_length:
                if current_chunk:
                    # `a_end` is how many segments from `current_chunk` go into the `A` (first) sentence.
                    a_end = 1
                    # if current chunk has more than 2 sentences, pick part of it `A` (first) sentence
                    if len(current_chunk) >= 2:
                        a_end = random.randint(1, len(current_chunk) - 1)
                    # token a
                    tokens_a = []
                    for j in range(a_end):
                        tokens_a.extend(current_chunk[j])

                    # token b
                    tokens_b = []
                    for j in range(a_end, len(current_chunk)):
                        tokens_b.extend(current_chunk[j])

                    if len(tokens_a) == 0 or len(tokens_b) == 0:
                        continue

                    # switch tokens_a and tokens_b randomly
                    if random.random() < 0.5:
                        is_next = False
                        tokens_a, tokens_b = tokens_b, tokens_a
                    else:
                        is_next = True

                    def truncate_seq_pair(tokens_a, tokens_b, max_num_tokens):
                        """Truncates a pair of sequences to a maximum sequence length."""
                        while True:
                            total_length = len(tokens_a) + len(tokens_b)
                            if total_length <= max_num_tokens:
                                break
                            trunc_tokens = tokens_a if len(tokens_a) > len(tokens_b) else tokens_b
                            if not (len(trunc_tokens) >= 1):
                                raise ValueError("Sequence length to be truncated must be no less than one")
                            # We want to sometimes truncate from the front and sometimes from the
                            # back to add more randomness and avoid biases.
                            if random.random() < 0.5:
                                del trunc_tokens[0]
                            else:
                                trunc_tokens.pop()

                    truncate_seq_pair(tokens_a, tokens_b, max_num_tokens)
                    if not (len(tokens_a) >= 1):
                        raise ValueError(f"Length of sequence a is {len(tokens_a)} which must be no less than 1")
                    if not (len(tokens_b) >= 1):
                        raise ValueError(f"Length of sequence b is {len(tokens_b)} which must be no less than 1")

                    # add special tokens
                    input_ids = tokenizer.build_inputs_with_special_tokens(tokens_a, tokens_b)
                    # add token type ids, 0 for sentence a, 1 for sentence b
                    token_type_ids = tokenizer.create_token_type_ids_from_sequences(tokens_a, tokens_b)

                    example = {
                        "input_ids": torch.tensor(input_ids, dtype=torch.long),
                        "token_type_ids": torch.tensor(token_type_ids, dtype=torch.long),
                        "sentence_order_label": torch.tensor(0 if is_next else 1, dtype=torch.long),
                    }
                    examples.append(example)
                current_chunk = []  # clear current chunk
                current_length = 0  # reset current text length
            i += 1  # go to next line
        return examples

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> Dict[str, torch.tensor]:


SUMMARY:
**Class Summary: LineByLineWithSOPTextDataset**

This class is designed to create a dataset for sentence order prediction (SOP) tasks. It reads file(s) containing article content and splits them into pairs of sentences (A and B). Each pair is then truncated to a specified length using a random probability threshold.

**Main Functionality:**

*   Reads file(s) containing article content from a specified directory
*   Splits the content into sentence pairs (A and B)
*   Trims each pair to a target sequence length with a probability threshold for short sequences
*   Creates examples with input IDs, token types, and sentence order labels

**Implementation Details:**

*   Utilizes Hugging Face's `PreTrainedTokenizer` for tokenization and text processing
*   Uses a buffer-driven approach to efficiently process long documents
*   Applies truncation logic to each pair using a probability threshold for short sequences

**Notable Dependencies:**

*   `transformers` library (Hugging Face) for tokenization and text processing
*   `torch` library for tensor operations and data types

**Key Class Elements:**

*   `__init__`: Initializes the dataset with a tokenizer, file directory, and block size
*   `create_examples_from_document`: Creates examples from a single document, including truncation logic
*   `__len__` and `__getitem__`: Return the total number of examples and retrieve individual examples by index


key: chroma:document
string_value:
CODE:

class TextDatasetForNextSentencePrediction(Dataset):
    """
    This will be superseded by a framework-agnostic approach soon.
    """

    def __init__(
        self,
        tokenizer: PreTrainedTokenizer,
        file_path: str,
        block_size: int,
        overwrite_cache=False,
        short_seq_probability=0.1,
        nsp_probability=0.5,
    ):
        warnings.warn(
            DEPRECATION_WARNING.format(
                "https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py"
            ),
            FutureWarning,
        )
        if not os.path.isfile(file_path):
            raise ValueError(f"Input file path {file_path} not found")

        self.short_seq_probability = short_seq_probability
        self.nsp_probability = nsp_probability

        directory, filename = os.path.split(file_path)
        cached_features_file = os.path.join(
            directory,
            f"cached_nsp_{tokenizer.__class__.__name__}_{block_size}_{filename}",
        )

        self.tokenizer = tokenizer

        # Make sure only the first process in distributed training processes the dataset,
        # and the others will use the cache.
        lock_path = cached_features_file + ".lock"

        # Input file format:
        # (1) One sentence per line. These should ideally be actual sentences, not
        # entire paragraphs or arbitrary spans of text. (Because we use the
        # sentence boundaries for the "next sentence prediction" task).
        # (2) Blank lines between documents. Document boundaries are needed so
        # that the "next sentence prediction" task doesn't span between documents.
        #
        # Example:
        # I am very happy.
        # Here is the second sentence.
        #
        # A new document.

        with FileLock(lock_path):
            if os.path.exists(cached_features_file) and not overwrite_cache:
                start = time.time()
                with open(cached_features_file, "rb") as handle:
                    self.examples = pickle.load(handle)
                logger.info(
                    f"Loading features from cached file {cached_features_file} [took %.3f s]", time.time() - start
                )
            else:
                logger.info(f"Creating features from dataset file at {directory}")

                self.documents = [[]]
                with open(file_path, encoding="utf-8") as f:
                    while True:
                        line = f.readline()
                        if not line:
                            break
                        line = line.strip()

                        # Empty lines are used as document delimiters
                        if not line and len(self.documents[-1]) != 0:
                            self.documents.append([])
                        tokens = tokenizer.tokenize(line)
                        tokens = tokenizer.convert_tokens_to_ids(tokens)
                        if tokens:
                            self.documents[-1].append(tokens)

                logger.info(f"Creating examples from {len(self.documents)} documents.")
                self.examples = []
                for doc_index, document in enumerate(self.documents):
                    self.create_examples_from_document(document, doc_index, block_size)

                start = time.time()
                with open(cached_features_file, "wb") as handle:
                    pickle.dump(self.examples, handle, protocol=pickle.HIGHEST_PROTOCOL)
                logger.info(
                    f"Saving features into cached file {cached_features_file} [took {time.time() - start:.3f} s]"
                )

    def create_examples_from_document(self, document: List[List[int]], doc_index: int, block_size: int):
        """Creates examples for a single document."""

        max_num_tokens = block_size - self.tokenizer.num_special_tokens_to_add(pair=True)

        # We *usually* want to fill up the entire sequence since we are padding
        # to `block_size` anyways, so short sequences are generally wasted
        # computation. However, we *sometimes*
        # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter
        # sequences to minimize the mismatch between pretraining and fine-tuning.
        # The `target_seq_length` is just a rough target however, whereas
        # `block_size` is a hard limit.
        target_seq_length = max_num_tokens
        if random.random() < self.short_seq_probability:
            target_seq_length = random.randint(2, max_num_tokens)

        current_chunk = []  # a buffer stored current working segments
        current_length = 0
        i = 0

        while i < len(document):
            segment = document[i]
            current_chunk.append(segment)
            current_length += len(segment)
            if i == len(document) - 1 or current_length >= target_seq_length:
                if current_chunk:
                    # `a_end` is how many segments from `current_chunk` go into the `A`
                    # (first) sentence.
                    a_end = 1
                    if len(current_chunk) >= 2:
                        a_end = random.randint(1, len(current_chunk) - 1)

                    tokens_a = []
                    for j in range(a_end):
                        tokens_a.extend(current_chunk[j])

                    tokens_b = []

                    if len(current_chunk) == 1 or random.random() < self.nsp_probability:
                        is_random_next = True
                        target_b_length = target_seq_length - len(tokens_a)

                        # This should rarely go for more than one iteration for large
                        # corpora. However, just to be careful, we try to make sure that
                        # the random document is not the same as the document
                        # we're processing.
                        for _ in range(10):
                            random_document_index = random.randint(0, len(self.documents) - 1)
                            if random_document_index != doc_index:
                                break

                        random_document = self.documents[random_document_index]
                        random_start = random.randint(0, len(random_document) - 1)
                        for j in range(random_start, len(random_document)):
                            tokens_b.extend(random_document[j])
                            if len(tokens_b) >= target_b_length:
                                break
                        # We didn't actually use these segments so we "put them back" so
                        # they don't go to waste.
                        num_unused_segments = len(current_chunk) - a_end
                        i -= num_unused_segments
                    # Actual next
                    else:
                        is_random_next = False
                        for j in range(a_end, len(current_chunk)):
                            tokens_b.extend(current_chunk[j])

                    if not (len(tokens_a) >= 1):
                        raise ValueError(f"Length of sequence a is {len(tokens_a)} which must be no less than 1")
                    if not (len(tokens_b) >= 1):
                        raise ValueError(f"Length of sequence b is {len(tokens_b)} which must be no less than 1")

                    # add special tokens
                    input_ids = self.tokenizer.build_inputs_with_special_tokens(tokens_a, tokens_b)
                    # add token type ids, 0 for sentence a, 1 for sentence b
                    token_type_ids = self.tokenizer.create_token_type_ids_from_sequences(tokens_a, tokens_b)

                    example = {
                        "input_ids": torch.tensor(input_ids, dtype=torch.long),
                        "token_type_ids": torch.tensor(token_type_ids, dtype=torch.long),
                        "next_sentence_label": torch.tensor(1 if is_random_next else 0, dtype=torch.long),
                    }

                    self.examples.append(example)

                current_chunk = []
                current_length = 0

            i += 1

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):


SUMMARY:
**Summary:**

`TextDatasetForNextSentencePrediction` is a custom dataset class for next sentence prediction tasks. It provides example pairs for training tasks. The dataset takes a text file and generates pair examples from the text.

*   **Main Functionality**: 
    *   Loads a text file containing one sentence per line.
    *   Applies tokenization, conversion to IDs, and creation of example pairs (A, B), including padding for sequences.
*   **Implementation Details**:
    *   Employs a buffer-based segmentation approach with `short_seq_probability` influencing sequence length variability.
    *   Utilizes a lock file mechanism for threaded training to ensure only one process processes the dataset.
    
**External Dependencies and Interfaces:**

*   Requires Hugging Face library (`transformers`)
*   Utilizes `PreTrainedTokenizer` from transformer
*   Uses `FileLock` for synchronization in multi-threaded computations


key: chroma:document
string_value:
CODE:

class Split(Enum):
    train = "train"
    dev = "dev"


SUMMARY:
**Summary**

This `Split` Enum class defines a set of enumeration values (`train`, `dev`, and `test`) representing different data splitting modes for a machine learning model.

**Key Points**

- **Primary Purpose**: Defines an enumeration to categorize data splits
- **Implementation Details**: None, the code only contains basic enum definition with no logic or methods.
- **Dependencies/Requirements**: None


key: chroma:document
string_value:
CODE:

class GlueDataset(Dataset):
    """
    This will be superseded by a framework-agnostic approach soon.
    """

    args: GlueDataTrainingArguments
    output_mode: str
    features: List[InputFeatures]

    def __init__(
        self,
        args: GlueDataTrainingArguments,
        tokenizer: PreTrainedTokenizerBase,
        limit_length: Optional[int] = None,
        mode: Union[str, Split] = Split.train,
        cache_dir: Optional[str] = None,
    ):
        warnings.warn(
            "This dataset will be removed from the library soon, preprocessing should be handled with the 🤗 Datasets "
            "library. You can have a look at this example script for pointers: "
            "https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py",
            FutureWarning,
        )
        self.args = args
        self.processor = glue_processors[args.task_name]()
        self.output_mode = glue_output_modes[args.task_name]
        if isinstance(mode, str):
            try:
                mode = Split[mode]
            except KeyError:
                raise KeyError("mode is not a valid split name")
        # Load data features from cache or dataset file
        cached_features_file = os.path.join(
            cache_dir if cache_dir is not None else args.data_dir,
            f"cached_{mode.value}_{tokenizer.__class__.__name__}_{args.max_seq_length}_{args.task_name}",
        )
        label_list = self.processor.get_labels()
        if args.task_name in ["mnli", "mnli-mm"] and tokenizer.__class__.__name__ in (
            "RobertaTokenizer",
            "RobertaTokenizerFast",
            "XLMRobertaTokenizer",
            "BartTokenizer",
            "BartTokenizerFast",
        ):
            # HACK(label indices are swapped in RoBERTa pretrained model)
            label_list[1], label_list[2] = label_list[2], label_list[1]
        self.label_list = label_list

        # Make sure only the first process in distributed training processes the dataset,
        # and the others will use the cache.
        lock_path = cached_features_file + ".lock"
        with FileLock(lock_path):
            if os.path.exists(cached_features_file) and not args.overwrite_cache:
                start = time.time()
                self.features = torch.load(cached_features_file)
                logger.info(
                    f"Loading features from cached file {cached_features_file} [took %.3f s]", time.time() - start
                )
            else:
                logger.info(f"Creating features from dataset file at {args.data_dir}")

                if mode == Split.dev:
                    examples = self.processor.get_dev_examples(args.data_dir)
                elif mode == Split.test:
                    examples = self.processor.get_test_examples(args.data_dir)
                else:
                    examples = self.processor.get_train_examples(args.data_dir)
                if limit_length is not None:
                    examples = examples[:limit_length]
                self.features = glue_convert_examples_to_features(
                    examples,
                    tokenizer,
                    max_length=args.max_seq_length,
                    label_list=label_list,
                    output_mode=self.output_mode,
                )
                start = time.time()
                torch.save(self.features, cached_features_file)
                # ^ This seems to take a lot of time so I want to investigate why and how we can improve.
                logger.info(
                    f"Saving features into cached file {cached_features_file} [took {time.time() - start:.3f} s]"
                )

    def __len__(self):
        return len(self.features)

    def __getitem__(self, i) -> InputFeatures:
        return self.features[i]

    def get_labels(self):


SUMMARY:
**Dataset Class Summary**

The `GlueDataset` class is a custom dataset implementation for the Hugging Face Transformers library. It extends the `Dataset` class and is specifically designed for tasks using the GLUE benchmark.

**Main Purpose and Functionality:**
This dataset class loads data from cache or dataset files, preprocesses it using a tokenizer, and stores the preprocessed features in memory. The purpose of this class is to provide an interface for training models on GLUE tasks.

**Key Implementation Details:**

*   It uses a `PreTrainedTokenizerBase` object to tokenize input data.
*   The `glue_processors` dictionary maps task names to their respective processor objects.
*   It handles caching of features in a file and allows for distributed training by storing metadata about the dataset.
*   The `glue_convert_examples_to_features` function is used to convert preprocessed examples into feature tensors.

**Notable Dependencies or Requirements:**

*   `Dataset`: Base class for datasets
*   `PreTrainedTokenizerBase`: tokenizer object to tokenize input data
*   `glue_processors`: dictionary mapping task names to their processor objects
*   `glue_convert_examples_to_features`: function to convert preprocessed examples into feature tensors

**Notable Variables and Interfaces:**

*   `self.args` : An instance of `GlueDataTrainingArguments` containing training arguments for the model.
*   `self.output_mode` : The output mode used during data preprocessing.
*   `self.processor` : A processor object specific to the task being performed.
*   `self.features` : The preprocessed features stored in memory.

**Interfaces and Methods:**

*   `__init__`: Initializes the dataset with training arguments, tokenizer, and cache directory.
*   `__len__`: Returns the length of the dataset by checking the dimensionality of the stored features.
*   `__getitem__`: Returns an example from the dataset at the specified index.
*   `get_labels`: Returns a list containing all label indices and text labels.

**Future Features and Changes:**
The code includes future warning about removal from the library soon. The GLUE dataset will be replaced by a more framework-agnostic approach in the near future.


key: chroma:document
string_value:
CODE:

def simple_accuracy(preds, labels):
    warnings.warn(DEPRECATION_WARNING, FutureWarning)
    requires_backends(simple_accuracy, "sklearn")


SUMMARY:
**Summary**

`simple_accuracy`: Calculates the mean accuracy of predictions by comparing them to true labels.

**Key Implementation Details**

* Computes a boolean mask where `preds` equals `labels`.
* Returns the mean value of this mask using `mean()`.
* Issues a deprecation warning if not running with compatible backend (`sklearn`).

**Requirements and Notable Dependencies**

* Requires `requires_backends` from `sklearn` ( backend is required for this function to run).


key: chroma:document
string_value:
CODE:

def acc_and_f1(preds, labels):
    warnings.warn(DEPRECATION_WARNING, FutureWarning)
    requires_backends(acc_and_f1, "sklearn")
    acc = simple_accuracy(preds, labels)
    f1 = f1_score(y_true=labels, y_pred=preds)
    return {
        "acc": acc,
        "f1": f1,
        "acc_and_f1": (acc + f1) / 2,


SUMMARY:
**Summary: `acc_and_f1` Function**

The `acc_and_f1` function calculates and returns metrics for accuracy and F1 score. It takes model predictions (`preds`) and ground truth labels (`labels`) as input.

**Main Purpose:** Calculate and return the average of accuracy and F1 score from given model predictions and labels.

**Key Implementation:**

* Calls `simple_accuracy` to compute accuracy
* Calls `f1_score` to compute F1 score
* Returns average of combined accuracy and F1 score

**Notable Dependencies/Requireds:**

* `simple_accuracy`
* `f1_score` from scikit-learn library (`sklearn`)


key: chroma:document
string_value:
CODE:

def pearson_and_spearman(preds, labels):
    warnings.warn(DEPRECATION_WARNING, FutureWarning)
    requires_backends(pearson_and_spearman, "sklearn")
    pearson_corr = pearsonr(preds, labels)[0]
    spearman_corr = spearmanr(preds, labels)[0]
    return {
        "pearson": pearson_corr,
        "spearmanr": spearman_corr,
        "corr": (pearson_corr + spearman_corr) / 2,


SUMMARY:
Here is a concise summary of the provided function:

**Function:** `pearson_and_spearman`

**Purpose:** Calculates and returns Pearson correlation coefficient, Spearman rank correlation coefficient, and their average.

**Key Implementation Details:**

* Computes Pearson correlation using `pearsonr(preds, labels)[0]`
* Computes Spearman rank correlation using `spearmanr(preds, labels)[0]`
* Returns a dictionary with correlations:
	+ "pearson": Pearson correlation coefficient
	+ "spearmanr": Spearman rank correlation coefficient
	+ "corr": Average of both correlation coefficients

**Notable Dependencies:**

* Uses `sklearn` backend (requires installation)
* Imports warning category `DEPRECATION_WARNING`
* Calls function `requires_backends` to enable deprecation warnings for future versions


key: chroma:document
string_value:
CODE:

def glue_compute_metrics(task_name, preds, labels):
    warnings.warn(DEPRECATION_WARNING, FutureWarning)
    requires_backends(glue_compute_metrics, "sklearn")
    assert len(preds) == len(labels), f"Predictions and labels have mismatched lengths {len(preds)} and {len(labels)}"
    if task_name == "cola":
        return {"mcc": matthews_corrcoef(labels, preds)}
    elif task_name == "sst-2":
        return {"acc": simple_accuracy(preds, labels)}
    elif task_name == "mrpc":
        return acc_and_f1(preds, labels)
    elif task_name == "sts-b":
        return pearson_and_spearman(preds, labels)
    elif task_name == "qqp":
        return acc_and_f1(preds, labels)
    elif task_name == "mnli":
        return {"mnli/acc": simple_accuracy(preds, labels)}
    elif task_name == "mnli-mm":
        return {"mnli-mm/acc": simple_accuracy(preds, labels)}
    elif task_name == "qnli":
        return {"acc": simple_accuracy(preds, labels)}
    elif task_name == "rte":
        return {"acc": simple_accuracy(preds, labels)}
    elif task_name == "wnli":
        return {"acc": simple_accuracy(preds, labels)}
    elif task_name == "hans":
        return {"acc": simple_accuracy(preds, labels)}
    else:


SUMMARY:
**Summary**

Glue Compute Metrics Function

**Purpose and Functionality**

Computes evaluation metrics for GLUE datasets. Takes in task name, predictions, and labels as input.

**Key Implementation Details**

* Computes different evaluation metrics depending on the task name.
	+ `matthews_corrcoef` and `simple_accuracy` are used for binary tasks (e.g.,cola, sentiment analysis).
	+ `acc_and_f1` is used for multi-class tasks (e.g., rte, ques answering).
* Returns a dictionary with evaluation metrics.

**Notable Dependencies or Requirements**

* Matplotlib and scipy for computations.
* sklearn's `compute`, which requires backends.

Note: The provided code uses outdated warnings (FutureWarning) and may require maintenance to be compatible with newer versions of scikit-learn.


key: chroma:document
string_value:
CODE:

def xnli_compute_metrics(task_name, preds, labels):
    warnings.warn(DEPRECATION_WARNING, FutureWarning)
    requires_backends(xnli_compute_metrics, "sklearn")
    if len(preds) != len(labels):
        raise ValueError(f"Predictions and labels have mismatched lengths {len(preds)} and {len(labels)}")
    if task_name == "xnli":
        return {"acc": simple_accuracy(preds, labels)}
    else:


SUMMARY:
**Summary**

`xnli_compute_metrics` function computes metrics for the XNLI task:

### Purpose and Functionality
Computes accuracy metric for XNLI dataset using `simple_accuracy`.

### Key Implementation Details
- Checks if predictions and labels have matching lengths.
- Raisese KeyError if unsupported task name is provided.

### Dependencies and Requirements
- Requires sklearn backend (deprecated warning).
- Supports XNLI task only.


key: chroma:document
string_value:
CODE:

def normalize_answer(s):
    """Lower text and remove punctuation, articles and extra whitespace."""

    def remove_articles(text):
        regex = re.compile(r"\b(a|an|the)\b", re.UNICODE)
        return re.sub(regex, " ", text)

    def white_space_fix(text):
        return " ".join(text.split())

    def remove_punc(text):
        exclude = set(string.punctuation)
        return "".join(ch for ch in text if ch not in exclude)

    def lower(text):
        return text.lower()



SUMMARY:
**Summary**

The `normalize_answer` function normalizes a given input string `s` by:

1. Removing articles ('a', 'an', 'the') using regular expression matching (`re.sub`)
2. Replacing non-alphanumeric characters with whitespace using `re.sub`
3. Removing leading/trailing whitespace using `" ".join(text.split())`
4. Converting to lowercase using `.lower()`

**Key Implementation Details**

* Utilizes `re` module for regular expression operations
* Uses sets to exclude specific punctuation characters (`string.punctuation`)
* Applies transformations in a pipeline manner (e.g., converting to lowercase before applying white space fix)

**Notable Dependencies or Requirements**

* [NLTK library](https://www.nltk.org/) for string processing functions (e.g., `split()`, `lower()`)
* [re module](https://docs.python.org/3/library/re.html) for regular expression operations

**External Dependencies and Interfaces**

None explicitly listed. However, the function is designed to work with input strings that only contain ASCII characters, implying compatibility with Python's built-in string handling functionality.


key: chroma:document
string_value:
CODE:

def get_tokens(s):
    if not s:
        return []


SUMMARY:
**Code Summary**

### Function Purpose and Functionality

The `get_tokens` function takes a string input `s` and returns a list of tokens derived from the string's whitespace-separated words. It handles empty strings by returning an empty list.

### Key Implementation Details

* Splits the input string into individual words using whitespace as delimiter.
* Applies normalization to the resulting string using the `normalize_answer` function, which is not defined in this snippet.

### Notable Dependencies and Requirements

* The existence of `normalize_answer` function, which is assumed to be defined elsewhere.
* No explicit external dependencies mentioned aside from potentially shared utility functions.


key: chroma:document
string_value:
CODE:

def compute_exact(a_gold, a_pred):


SUMMARY:
**Summary:**

* **Function Name:** `compute_exact`
* **Primary Purpose:** Evaluate the exactness of two numerical answers by comparing their normalized values.

**Key Implementation Details:**
- Uses the `normalize_answer` function, which is not shown in this code snippet.

**Method Signature and Interfaces:**
- The `compute_exact` function takes two arguments `a_gold` and `a_pred` and returns an integer value indicating exactness (1 for true, 0 for false).

**Notable Dependencies and Requirements:**
- References the external `normalize_answer` function.

This summary focuses on the essential details of the code chunk, omitting unnecessary information.


key: chroma:document
string_value:
CODE:

def compute_f1(a_gold, a_pred):
    gold_toks = get_tokens(a_gold)
    pred_toks = get_tokens(a_pred)
    common = collections.Counter(gold_toks) & collections.Counter(pred_toks)
    num_same = sum(common.values())
    if len(gold_toks) == 0 or len(pred_toks) == 0:
        # If either is no-answer, then F1 is 1 if they agree, 0 otherwise
        return int(gold_toks == pred_toks)
    if num_same == 0:
        return 0
    precision = 1.0 * num_same / len(pred_toks)
    recall = 1.0 * num_same / len(gold_toks)
    f1 = (2 * precision * recall) / (precision + recall)


SUMMARY:
**Function Summary**

`compute_f1(a_gold, a_pred)`: Evaluates the F1 score between two prediction results `a_gold` and `a_pred`.

**Main Purpose and Functionality**
Calculates F1 score, which is harmonic mean of precision and recall, to measure accuracy of entity recognition.

### Key Implementation Details

* Uses `collections.Counter` to find common tokens and calculate overlap (`num_same`).
* Calculates precision and recall ratios (`precision`, `recall`) using exact token counts.
* Computes F1 score as `(2 * precision * recall) / (precision + recall)`.

### Notable Dependencies or Requirements

* `get_tokens(a_gold)` and `get_tokens(a_pred)` functions (not shown in excerpt).
* External dependency: `collections`.

### Method Signatures and Interfaces
`compute_f1`: takes two input arguments (`a_gold`, `a_pred`) with no explicit interfaces.


key: chroma:document
string_value:
CODE:

def get_raw_scores(examples, preds):
    """
    Computes the exact and f1 scores from the examples and the model predictions
    """
    exact_scores = {}
    f1_scores = {}

    for example in examples:
        qas_id = example.qas_id
        gold_answers = [answer["text"] for answer in example.answers if normalize_answer(answer["text"])]

        if not gold_answers:
            # For unanswerable questions, only correct answer is empty string
            gold_answers = [""]

        if qas_id not in preds:
            print(f"Missing prediction for {qas_id}")
            continue

        prediction = preds[qas_id]
        exact_scores[qas_id] = max(compute_exact(a, prediction) for a in gold_answers)
        f1_scores[qas_id] = max(compute_f1(a, prediction) for a in gold_answers)



SUMMARY:
**Summary:**

`get_raw_scores` function computes exact and F1 scores from model predictions.

**Key Details:**

- Calculates exact and F1 scores using `compute_exact` and `compute_f1` functions for each example's gold answers.
- Handles unanswerable questions by treating the only predicted answer as an empty string.
- Returns two dictionaries: `exact_scores` and `f1_scores`.

**Implementation Notes:**

- Uses a dictionary to store scores for each question (qas_id).
- Uses list comprehensions to calculate scores for each gold answer.

**Dependencies/Interfaces:**

- Assumes existence of `normalize_answer`, `compute_exact`, and `compute_f1` functions.
- Requires external dependencies on the `text` data structure and `dict` data type.


key: chroma:document
string_value:
CODE:

def apply_no_ans_threshold(scores, na_probs, qid_to_has_ans, na_prob_thresh):
    new_scores = {}
    for qid, s in scores.items():
        pred_na = na_probs[qid] > na_prob_thresh
        if pred_na:
            new_scores[qid] = float(not qid_to_has_ans[qid])
        else:
            new_scores[qid] = s


SUMMARY:
**Summary: Application of No Answer Threshold Function**

The `apply_no_ans_threshold` function applies a threshold-based approach to determine predicted non-answers (NA) based on provided probabilities and confidence scores.

**Key Functionality:**

*   Returns a new dictionary (`new_scores`) with updated NA predictions.
*   Predicts NA if the probability is above the specified threshold (`na_prob_thresh`).
*   Outputs `1.0` as a predicted score for NA when true, and the original score (`s`) otherwise.

**Notable Implementation Details:**

*   Uses a dictionary comprehension to create the new scores dictionary.
*   Applies conditional assignment based on probability thresholds.
*   Relies on external data structures: `[scores]`, `[na_probs]`, `[qid_to_has_ans]`, and `[na_prob_thresh]`.

**Assumptions:**

*   Requires access to predefined probabilities (`na_probs`) and confidence score dictionary (`qid_to_has_ans`).
*   Applies a threshold-based approach for probability (`na_prob_thresh`).


key: chroma:document
string_value:
CODE:

def make_eval_dict(exact_scores, f1_scores, qid_list=None):
    if not qid_list:
        total = len(exact_scores)
        return collections.OrderedDict(
            [
                ("exact", 100.0 * sum(exact_scores.values()) / total),
                ("f1", 100.0 * sum(f1_scores.values()) / total),
                ("total", total),
            ]
        )
    else:
        total = len(qid_list)
        return collections.OrderedDict(
            [
                ("exact", 100.0 * sum(exact_scores[k] for k in qid_list) / total),
                ("f1", 100.0 * sum(f1_scores[k] for k in qid_list) / total),
                ("total", total),
            ]


SUMMARY:
**Summary:**

This function, `make_eval_dict`, calculates and returns an evaluation dictionary that contains metrics for exact scores, F1 scores, and the total number of QIDs. It takes two input dictionaries (`exact_scores` and `f1_scores`) and an optional list of QIDs (`qid_list`). The calculation of exact/F1 scores is normalized to account for the specified list of QIDs or all QIDs if not provided.

**Key Implementation Details:**

* Uses `collections.OrderedDict` for efficient dictionary creation.
* Normalizes scores by summing values over a specified set of keys (QIDs) and dividing by the total count.

**Notable Dependencies/Requirements:**

* Assumes the presence of two dictionaries (`exact_scores` and `f1_scores`) with string keys.
* Optionally uses an external list or dictionary (`qid_list`) to specify the scope for score calculations.


key: chroma:document
string_value:
CODE:

def merge_eval(main_eval, new_eval, prefix):
    for k in new_eval:


SUMMARY:
**Concise Summary**

`merge_eval` merges two evaluation functions (`main_eval` and `new_eval`) by extending the former with prefixes to new values. It:

* Takes three parameters: `main_eval`, `new_eval`, and `prefix`.
* Iterates over keys in `new_eval` and updates `main_eval` with prefixed versions of their values.

**Details**

* Implemented using a `for` loop, iterating over all keys (`k`) in `new_eval`.
* Updates `main_eval` dictionary with new key-value pairs where the key is constructed by concatenating `prefix_` with original key (`k`).
* No specific algorithm or pattern used; simple extension of dictionary.

**Notable Dependencies**

* None explicitly mentioned.

**Interfaces/Inherited Traits**

* Dictionary interface (Python's built-in).

Note: The provided code assumes Python 3.5+ syntax, and its functionality mirrors that of the `update()` method on dictionaries in Python.


key: chroma:document
string_value:
CODE:

def find_best_thresh_v2(preds, scores, na_probs, qid_to_has_ans):
    num_no_ans = sum(1 for k in qid_to_has_ans if not qid_to_has_ans[k])
    cur_score = num_no_ans
    best_score = cur_score
    best_thresh = 0.0
    qid_list = sorted(na_probs, key=lambda k: na_probs[k])
    for i, qid in enumerate(qid_list):
        if qid not in scores:
            continue
        if qid_to_has_ans[qid]:
            diff = scores[qid]
        else:
            if preds[qid]:
                diff = -1
            else:
                diff = 0
        cur_score += diff
        if cur_score > best_score:
            best_score = cur_score
            best_thresh = na_probs[qid]

    has_ans_score, has_ans_cnt = 0, 0
    for qid in qid_list:
        if not qid_to_has_ans[qid]:
            continue
        has_ans_cnt += 1

        if qid not in scores:
            continue
        has_ans_score += scores[qid]



SUMMARY:
**Concise Summary: `find_best_thresh_v2` Function**

**Purpose and Functionality**
 finds the optimal threshold for detecting answers in a set of predictions.

**Key Implementation Details**

* Sorts candidate questions (qid) by their uncertainty scores (na_probs).
* Iterates over sorted qid list, calculating a cumulative score based on predicted answer probabilities.
* Updates best score and corresponding threshold when a higher score is found.
* Counts the total number of questions with answers (has_ans_cnt) and calculates its weighted average score.

**Input Dependencies**

* `preds`: predicts
* `scores`: scores dictionary
* `na_probs`: uncertainty scores for qid list

**Return Value**

* A tuple containing:
	+ Optimized confidence: 100.0 * best_score / len(scores)
	+ Optimal threshold (best_thresh): na_probs[qid]
	+ Has answer ratio: has_ans_score / has_ans_cnt


key: chroma:document
string_value:
CODE:

def find_all_best_thresh_v2(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans):
    best_exact, exact_thresh, has_ans_exact = find_best_thresh_v2(preds, exact_raw, na_probs, qid_to_has_ans)
    best_f1, f1_thresh, has_ans_f1 = find_best_thresh_v2(preds, f1_raw, na_probs, qid_to_has_ans)
    main_eval["best_exact"] = best_exact
    main_eval["best_exact_thresh"] = exact_thresh
    main_eval["best_f1"] = best_f1
    main_eval["best_f1_thresh"] = f1_thresh
    main_eval["has_ans_exact"] = has_ans_exact


SUMMARY:
**Summary**

The `find_all_best_thresh_v2` function computes and aggregates several evaluation metrics (F1 scores, threshold values) for a given task. 

* **Computes**: Finds the best threshold for two evaluation metrics: exactness and F1 score, using individual and combined versions of these metrics.
* **Returns**: Stores the computed values in a dictionary `main_eval` with precise variable names.

**Key Implementation Details**

- Utilizes `find_best_thresh_v2` function to compute threshold values
- Uses `qedict` structure (`qd_to_has_ans`) for evaluation process

**Notable Dependencies or Requirements**

None identified


key: chroma:document
string_value:
CODE:

def find_best_thresh(preds, scores, na_probs, qid_to_has_ans):
    num_no_ans = sum(1 for k in qid_to_has_ans if not qid_to_has_ans[k])
    cur_score = num_no_ans
    best_score = cur_score
    best_thresh = 0.0
    qid_list = sorted(na_probs, key=lambda k: na_probs[k])
    for _, qid in enumerate(qid_list):
        if qid not in scores:
            continue
        if qid_to_has_ans[qid]:
            diff = scores[qid]
        else:
            if preds[qid]:
                diff = -1
            else:
                diff = 0
        cur_score += diff
        if cur_score > best_score:
            best_score = cur_score
            best_thresh = na_probs[qid]


SUMMARY:
**Function Summary: `find_best_thresh`**

**Purpose:** Determine the threshold for annotating questions with answers based on prediction scores and question likelihood of not having an answer.

**Key Implementation Details:**

* Iterates through sorted likelihoods of not having an answer (na_probs) to prioritize high-probability questions.
* Updates a running score (`cur_score`) by considering the difference between prediction scores and question likelihoods (represented by `scores[qid]`).
* Returns two values:
	+ The optimized threshold score (adjusted for the entire dataset size).
	+ The best threshold likelihood of not having an answer.

**Dependencies and Requirements:**

* `preds`
* `scores`
* `na_probs`
* `qid_to_has_ans`
* External dependencies:
	+ Assumption of a sorted dictionary (`na_probs`)

**Class/Function/Variable Names:** None

Note that I've removed extraneous details to provide a concise summary, while still highlighting the key aspects of the function.


key: chroma:document
string_value:
CODE:

def find_all_best_thresh(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans):
    best_exact, exact_thresh = find_best_thresh(preds, exact_raw, na_probs, qid_to_has_ans)
    best_f1, f1_thresh = find_best_thresh(preds, f1_raw, na_probs, qid_to_has_ans)

    main_eval["best_exact"] = best_exact
    main_eval["best_exact_thresh"] = exact_thresh
    main_eval["best_f1"] = best_f1


SUMMARY:
**Summary**

* Function `find_all_best_thresh` calculates and stores the best threshold values for exactness (`exact_thresh`) and F1 score (`f1_thresh`) from given prediction data.
* Takes in:
	+ `main_eval`: Main evaluation dictionary
	+ `preds`, `exact_raw`, `na_probs`, `qid_to_has_ans`: input prediction data, raw labels, etc.
* Returns: None (modifies `main_eval`)

**Key Details**

* Utilizes two separate invocations of `find_best_thresh` to compute best thresholds for exactness and F1 score.
* Stores results in `main_eval` dictionary.

**Notable Dependencies**

* Input classes/functions:
	+ `find_best_thresh`
	+ `preds`, `exact_raw`, `na_probs`, `qid_to_has_ans`
* Output modification:
	+ Modifies `main_eval` dictionary


key: chroma:document
string_value:
CODE:

def squad_evaluate(examples, preds, no_answer_probs=None, no_answer_probability_threshold=1.0):
    qas_id_to_has_answer = {example.qas_id: bool(example.answers) for example in examples}
    has_answer_qids = [qas_id for qas_id, has_answer in qas_id_to_has_answer.items() if has_answer]
    no_answer_qids = [qas_id for qas_id, has_answer in qas_id_to_has_answer.items() if not has_answer]

    if no_answer_probs is None:
        no_answer_probs = {k: 0.0 for k in preds}

    exact, f1 = get_raw_scores(examples, preds)

    exact_threshold = apply_no_ans_threshold(
        exact, no_answer_probs, qas_id_to_has_answer, no_answer_probability_threshold
    )
    f1_threshold = apply_no_ans_threshold(f1, no_answer_probs, qas_id_to_has_answer, no_answer_probability_threshold)

    evaluation = make_eval_dict(exact_threshold, f1_threshold)

    if has_answer_qids:
        has_ans_eval = make_eval_dict(exact_threshold, f1_threshold, qid_list=has_answer_qids)
        merge_eval(evaluation, has_ans_eval, "HasAns")

    if no_answer_qids:
        no_ans_eval = make_eval_dict(exact_threshold, f1_threshold, qid_list=no_answer_qids)
        merge_eval(evaluation, no_ans_eval, "NoAns")

    if no_answer_probs:
        find_all_best_thresh(evaluation, preds, exact, f1, no_answer_probs, qas_id_to_has_answer)



SUMMARY:
**Concise Summary**

```python
def squad_evaluate(examples, preds):
  """
  Evaluates the SQuAD model's performance on given examples and predictions.

  Parameters:
  - examples (list): SQuAD example instances
  - preds (dict): prediction dictionaries for each example's qas_id

  Returns:
  - evaluation (dict): merged evaluation metrics for HasAns, NoAns, and threshold-based scores
  """

  # Process examples and predictions
  qas_ids = {example.qas_id: bool(example.answers) for example in examples}
  has_answer_qids = [qid for qid, has_answer in qas_ids.items() if has_answer]
  no_answer_qids = [qid for qid, has_answer in qas_ids.items() if not has_answer]

  # Compute raw scores and thresholds
  exact_score, f1_score = get_raw_scores(examples, preds)
  exact_threshold = apply_no_ans_threshold(exact_score, preds, qas_ids, no_answer_probability_threshold=1.0)

  # Construct and merge evaluation dictionaries
  evaluation = make_eval_dict(exact_threshold, f1_score)
  if has_answer_qids:
    has_ans_eval = make_eval_dict(exact_threshold, f1_score, qid_list=has_answer_qids)
    merge_eval(evaluation, has_ans_eval, "HasAns")
  if no_answer_qids:
    no_ans_eval = make_eval_dict(exact_threshold, f1_score, qid_list=no_answer_qids)
    merge_eval(evaluation, no_ans_eval, "NoAns")

  # Find best threshold for scores
  if preds:
    find_all_best_thresh(evaluation, preds, exact_score, f1_score, qas_ids)

  return evaluation
```
**Key Points:**

* Evaluates SQuAD model performance on given examples and predictions.
* Returns a merged dictionary with HasAns, NoAns, and threshold-based scores.
* Processes examples, predictions, and computes raw scores and thresholds using `get_raw_scores` and `apply_no_ans_threshold`.
* Constructs and merges evaluation dictionaries for HasAns and NoAns scores using `make_eval_dict` and `merge_eval`.
* Finds best threshold for scores using `find_all_best_thresh`.


key: chroma:document
string_value:
CODE:

def get_final_text(pred_text, orig_text, do_lower_case, verbose_logging=False):
    """Project the tokenized prediction back to the original text."""

    # When we created the data, we kept track of the alignment between original
    # (whitespace tokenized) tokens and our WordPiece tokenized tokens. So
    # now `orig_text` contains the span of our original text corresponding to the
    # span that we predicted.
    #
    # However, `orig_text` may contain extra characters that we don't want in
    # our prediction.
    #
    # For example, let's say:
    #   pred_text = steve smith
    #   orig_text = Steve Smith's
    #
    # We don't want to return `orig_text` because it contains the extra "'s".
    #
    # We don't want to return `pred_text` because it's already been normalized
    # (the SQuAD eval script also does punctuation stripping/lower casing but
    # our tokenizer does additional normalization like stripping accent
    # characters).
    #
    # What we really want to return is "Steve Smith".
    #
    # Therefore, we have to apply a semi-complicated alignment heuristic between
    # `pred_text` and `orig_text` to get a character-to-character alignment. This
    # can fail in certain cases in which case we just return `orig_text`.

    def _strip_spaces(text):
        ns_chars = []
        ns_to_s_map = collections.OrderedDict()
        for i, c in enumerate(text):
            if c == " ":
                continue
            ns_to_s_map[len(ns_chars)] = i
            ns_chars.append(c)
        ns_text = "".join(ns_chars)
        return (ns_text, ns_to_s_map)

    # We first tokenize `orig_text`, strip whitespace from the result
    # and `pred_text`, and check if they are the same length. If they are
    # NOT the same length, the heuristic has failed. If they are the same
    # length, we assume the characters are one-to-one aligned.
    tokenizer = BasicTokenizer(do_lower_case=do_lower_case)

    tok_text = " ".join(tokenizer.tokenize(orig_text))

    start_position = tok_text.find(pred_text)
    if start_position == -1:
        if verbose_logging:
            logger.info(f"Unable to find text: '{pred_text}' in '{orig_text}'")
        return orig_text
    end_position = start_position + len(pred_text) - 1

    (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text)
    (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text)

    if len(orig_ns_text) != len(tok_ns_text):
        if verbose_logging:
            logger.info(f"Length not equal after stripping spaces: '{orig_ns_text}' vs '{tok_ns_text}'")
        return orig_text

    # We then project the characters in `pred_text` back to `orig_text` using
    # the character-to-character alignment.
    tok_s_to_ns_map = {}
    for i, tok_index in tok_ns_to_s_map.items():
        tok_s_to_ns_map[tok_index] = i

    orig_start_position = None
    if start_position in tok_s_to_ns_map:
        ns_start_position = tok_s_to_ns_map[start_position]
        if ns_start_position in orig_ns_to_s_map:
            orig_start_position = orig_ns_to_s_map[ns_start_position]

    if orig_start_position is None:
        if verbose_logging:
            logger.info("Couldn't map start position")
        return orig_text

    orig_end_position = None
    if end_position in tok_s_to_ns_map:
        ns_end_position = tok_s_to_ns_map[end_position]
        if ns_end_position in orig_ns_to_s_map:
            orig_end_position = orig_ns_to_s_map[ns_end_position]

    if orig_end_position is None:
        if verbose_logging:
            logger.info("Couldn't map end position")
        return orig_text

    output_text = orig_text[orig_start_position : (orig_end_position + 1)]


SUMMARY:
**Summary:**

`get_final_text` function takes in tokenized prediction and original text, along with parameters for lowercasing and verbose logging. It projects the predicted token back to the original text while preserving character-to-character alignment.

**Main Purpose and Functionality:**

The primary purpose is to align the predicted token with the corresponding characters in the original text to produce a normalized and accurate output.

**Key Implementation Details:**

*   Tokenizes input texts using `BasicTokenizer`
*   Applies character-to-character alignment heuristic between `pred_text` and `orig_text`
*   Uses `_strip_spaces` function to remove whitespace from the result

**Notable Dependencies or Requirements:**

*   `BasicTokenizer` class from Hugging Face Transformers library
*   `collections.OrderedDict` for mapping characters in original text

The code provides a complex alignment process between two tokens from different sources, handling discrepancies and edge cases.


key: chroma:document
string_value:
CODE:

def _get_best_indexes(logits, n_best_size):
    """Get the n-best logits from a list."""
    index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True)

    best_indexes = []
    for i in range(len(index_and_score)):
        if i >= n_best_size:
            break
        best_indexes.append(index_and_score[i][0])


SUMMARY:
**Summary:**

`_get_best_indexes` function:

* Returns the top `n_best_size` indexes from a sorted list of logits with corresponding scores.
* Sorts logits in descending order by score using `enumerate` and lambda function.

**Key Details:**

* Implementation details:
	+ Use of `sorted` and `enumerate` to sort logos by score.
	+ Filtering of top `n_best_size` elements using a for loop.
* Notable dependencies/inherited interfaces:
	+ None

**Method Signature:**
```python
def _get_best_indexes(logits, n_best_size):
    ...
```

This function appears to be part of a natural language processing or machine learning application. The purpose is likely to retrieve the top-scoring indices (e.g., sentence IDs) from a logged scoring output.


key: chroma:document
string_value:
CODE:

def _compute_softmax(scores):
    """Compute softmax probability over raw logits."""
    if not scores:
        return []

    max_score = None
    for score in scores:
        if max_score is None or score > max_score:
            max_score = score

    exp_scores = []
    total_sum = 0.0
    for score in scores:
        x = math.exp(score - max_score)
        exp_scores.append(x)
        total_sum += x

    probs = []
    for score in exp_scores:
        probs.append(score / total_sum)


SUMMARY:
**Summary**

`_compute_softmax(scores)` computes softmax probability over raw logits. This function normalizes the input `scores` to produce a probability distribution, where each element is the ratio of its corresponding logit's exponential value to the sum of all exponential values.

**Key Implementation Details**

* Computation of maximum score (`max_score`) to normalize scores.
* Exponential calculation (`exp_scores`) and calculation of total sum.
* Probability computation by dividing logarithmic probability values (`probs = exp_scores / total_sum`).

**Notable Dependencies/Requirements**

* `math` module (specifically, the `exp` function).


key: chroma:document
string_value:
CODE:

def compute_predictions_logits(
    all_examples,
    all_features,
    all_results,
    n_best_size,
    max_answer_length,
    do_lower_case,
    output_prediction_file,
    output_nbest_file,
    output_null_log_odds_file,
    verbose_logging,
    version_2_with_negative,
    null_score_diff_threshold,
    tokenizer,
):
    """Write final predictions to the json file and log-odds of null if needed."""
    if output_prediction_file:
        logger.info(f"Writing predictions to: {output_prediction_file}")
    if output_nbest_file:
        logger.info(f"Writing nbest to: {output_nbest_file}")
    if output_null_log_odds_file and version_2_with_negative:
        logger.info(f"Writing null_log_odds to: {output_null_log_odds_file}")

    example_index_to_features = collections.defaultdict(list)
    for feature in all_features:
        example_index_to_features[feature.example_index].append(feature)

    unique_id_to_result = {}
    for result in all_results:
        unique_id_to_result[result.unique_id] = result

    _PrelimPrediction = collections.namedtuple(  # pylint: disable=invalid-name
        "PrelimPrediction", ["feature_index", "start_index", "end_index", "start_logit", "end_logit"]
    )

    all_predictions = collections.OrderedDict()
    all_nbest_json = collections.OrderedDict()
    scores_diff_json = collections.OrderedDict()

    for example_index, example in enumerate(all_examples):
        features = example_index_to_features[example_index]

        prelim_predictions = []
        # keep track of the minimum score of null start+end of position 0
        score_null = 1000000  # large and positive
        min_null_feature_index = 0  # the paragraph slice with min null score
        null_start_logit = 0  # the start logit at the slice with min null score
        null_end_logit = 0  # the end logit at the slice with min null score
        for feature_index, feature in enumerate(features):
            result = unique_id_to_result[feature.unique_id]
            start_indexes = _get_best_indexes(result.start_logits, n_best_size)
            end_indexes = _get_best_indexes(result.end_logits, n_best_size)
            # if we could have irrelevant answers, get the min score of irrelevant
            if version_2_with_negative:
                feature_null_score = result.start_logits[0] + result.end_logits[0]
                if feature_null_score < score_null:
                    score_null = feature_null_score
                    min_null_feature_index = feature_index
                    null_start_logit = result.start_logits[0]
                    null_end_logit = result.end_logits[0]
            for start_index in start_indexes:
                for end_index in end_indexes:
                    # We could hypothetically create invalid predictions, e.g., predict
                    # that the start of the span is in the question. We throw out all
                    # invalid predictions.
                    if start_index >= len(feature.tokens):
                        continue
                    if end_index >= len(feature.tokens):
                        continue
                    if start_index not in feature.token_to_orig_map:
                        continue
                    if end_index not in feature.token_to_orig_map:
                        continue
                    if not feature.token_is_max_context.get(start_index, False):
                        continue
                    if end_index < start_index:
                        continue
                    length = end_index - start_index + 1
                    if length > max_answer_length:
                        continue
                    prelim_predictions.append(
                        _PrelimPrediction(
                            feature_index=feature_index,
                            start_index=start_index,
                            end_index=end_index,
                            start_logit=result.start_logits[start_index],
                            end_logit=result.end_logits[end_index],
                        )
                    )
        if version_2_with_negative:
            prelim_predictions.append(
                _PrelimPrediction(
                    feature_index=min_null_feature_index,
                    start_index=0,
                    end_index=0,
                    start_logit=null_start_logit,
                    end_logit=null_end_logit,
                )
            )
        prelim_predictions = sorted(prelim_predictions, key=lambda x: (x.start_logit + x.end_logit), reverse=True)

        _NbestPrediction = collections.namedtuple(  # pylint: disable=invalid-name
            "NbestPrediction", ["text", "start_logit", "end_logit"]
        )

        seen_predictions = {}
        nbest = []
        for pred in prelim_predictions:
            if len(nbest) >= n_best_size:
                break
            feature = features[pred.feature_index]
            if pred.start_index > 0:  # this is a non-null prediction
                tok_tokens = feature.tokens[pred.start_index : (pred.end_index + 1)]
                orig_doc_start = feature.token_to_orig_map[pred.start_index]
                orig_doc_end = feature.token_to_orig_map[pred.end_index]
                orig_tokens = example.doc_tokens[orig_doc_start : (orig_doc_end + 1)]

                tok_text = tokenizer.convert_tokens_to_string(tok_tokens)

                # tok_text = " ".join(tok_tokens)
                #
                # # De-tokenize WordPieces that have been split off.
                # tok_text = tok_text.replace(" ##", "")
                # tok_text = tok_text.replace("##", "")

                # Clean whitespace
                tok_text = tok_text.strip()
                tok_text = " ".join(tok_text.split())
                orig_text = " ".join(orig_tokens)

                final_text = get_final_text(tok_text, orig_text, do_lower_case, verbose_logging)
                if final_text in seen_predictions:
                    continue

                seen_predictions[final_text] = True
            else:
                final_text = ""
                seen_predictions[final_text] = True

            nbest.append(_NbestPrediction(text=final_text, start_logit=pred.start_logit, end_logit=pred.end_logit))
        # if we didn't include the empty option in the n-best, include it
        if version_2_with_negative:
            if "" not in seen_predictions:
                nbest.append(_NbestPrediction(text="", start_logit=null_start_logit, end_logit=null_end_logit))

            # In very rare edge cases we could only have single null prediction.
            # So we just create a nonce prediction in this case to avoid failure.
            if len(nbest) == 1:
                nbest.insert(0, _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))

        # In very rare edge cases we could have no valid predictions. So we
        # just create a nonce prediction in this case to avoid failure.
        if not nbest:
            nbest.append(_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))

        if len(nbest) < 1:
            raise ValueError("No valid predictions")

        total_scores = []
        best_non_null_entry = None
        for entry in nbest:
            total_scores.append(entry.start_logit + entry.end_logit)
            if not best_non_null_entry:
                if entry.text:
                    best_non_null_entry = entry

        probs = _compute_softmax(total_scores)

        nbest_json = []
        for i, entry in enumerate(nbest):
            output = collections.OrderedDict()
            output["text"] = entry.text
            output["probability"] = probs[i]
            output["start_logit"] = entry.start_logit
            output["end_logit"] = entry.end_logit
            nbest_json.append(output)

        if len(nbest_json) < 1:
            raise ValueError("No valid predictions")

        if not version_2_with_negative:
            all_predictions[example.qas_id] = nbest_json[0]["text"]
        else:
            # predict "" iff the null score - the score of best non-null > threshold
            score_diff = score_null - best_non_null_entry.start_logit - (best_non_null_entry.end_logit)
            scores_diff_json[example.qas_id] = score_diff
            if score_diff > null_score_diff_threshold:
                all_predictions[example.qas_id] = ""
            else:
                all_predictions[example.qas_id] = best_non_null_entry.text
        all_nbest_json[example.qas_id] = nbest_json

    if output_prediction_file:
        with open(output_prediction_file, "w") as writer:
            writer.write(json.dumps(all_predictions, indent=4) + "\n")

    if output_nbest_file:
        with open(output_nbest_file, "w") as writer:
            writer.write(json.dumps(all_nbest_json, indent=4) + "\n")

    if output_null_log_odds_file and version_2_with_negative:
        with open(output_null_log_odds_file, "w") as writer:
            writer.write(json.dumps(scores_diff_json, indent=4) + "\n")



SUMMARY:
The given function `compute_predictions_logits` appears to be part of a natural language processing (NLP) model that generates predictions for a quiz-based or question-answering task. Here's a concise summary:

This function takes in various inputs, including examples, features, results, and output file handles. It processes these inputs to generate predictions in the format of valid answers with their corresponding probabilities.

The process involves the following steps:

1. Identifying irrelevant answers by checking if any span starts before the previous one.
2. Generating initial prediction sets based on input data
3. Sorting and ranking predictions (initial set) according to weighted scores calculated from logit values in these rankings
4. Identifying empty predictions that fall below certain threshold.

Upon these steps, final output is constructed from ordered lists of predicted text and calculated probabilities used for the scoring model

Functions can return `all_predictions` dictionary which holds one response text per example based on best log-likelihood value and associated probability for every single example


key: chroma:document
string_value:
CODE:

def compute_predictions_log_probs(
    all_examples,
    all_features,
    all_results,
    n_best_size,
    max_answer_length,
    output_prediction_file,
    output_nbest_file,
    output_null_log_odds_file,
    start_n_top,
    end_n_top,
    version_2_with_negative,
    tokenizer,
    verbose_logging,
):
    """
    XLNet write prediction logic (more complex than Bert's). Write final predictions to the json file and log-odds of
    null if needed.

    Requires utils_squad_evaluate.py
    """
    _PrelimPrediction = collections.namedtuple(  # pylint: disable=invalid-name
        "PrelimPrediction", ["feature_index", "start_index", "end_index", "start_log_prob", "end_log_prob"]
    )

    _NbestPrediction = collections.namedtuple(  # pylint: disable=invalid-name
        "NbestPrediction", ["text", "start_log_prob", "end_log_prob"]
    )

    logger.info(f"Writing predictions to: {output_prediction_file}")

    example_index_to_features = collections.defaultdict(list)
    for feature in all_features:
        example_index_to_features[feature.example_index].append(feature)

    unique_id_to_result = {}
    for result in all_results:
        unique_id_to_result[result.unique_id] = result

    all_predictions = collections.OrderedDict()
    all_nbest_json = collections.OrderedDict()
    scores_diff_json = collections.OrderedDict()

    for example_index, example in enumerate(all_examples):
        features = example_index_to_features[example_index]

        prelim_predictions = []
        # keep track of the minimum score of null start+end of position 0
        score_null = 1000000  # large and positive

        for feature_index, feature in enumerate(features):
            result = unique_id_to_result[feature.unique_id]

            cur_null_score = result.cls_logits

            # if we could have irrelevant answers, get the min score of irrelevant
            score_null = min(score_null, cur_null_score)

            for i in range(start_n_top):
                for j in range(end_n_top):
                    start_log_prob = result.start_logits[i]
                    start_index = result.start_top_index[i]

                    j_index = i * end_n_top + j

                    end_log_prob = result.end_logits[j_index]
                    end_index = result.end_top_index[j_index]

                    # We could hypothetically create invalid predictions, e.g., predict
                    # that the start of the span is in the question. We throw out all
                    # invalid predictions.
                    if start_index >= feature.paragraph_len - 1:
                        continue
                    if end_index >= feature.paragraph_len - 1:
                        continue

                    if not feature.token_is_max_context.get(start_index, False):
                        continue
                    if end_index < start_index:
                        continue
                    length = end_index - start_index + 1
                    if length > max_answer_length:
                        continue

                    prelim_predictions.append(
                        _PrelimPrediction(
                            feature_index=feature_index,
                            start_index=start_index,
                            end_index=end_index,
                            start_log_prob=start_log_prob,
                            end_log_prob=end_log_prob,
                        )
                    )

        prelim_predictions = sorted(
            prelim_predictions, key=lambda x: (x.start_log_prob + x.end_log_prob), reverse=True
        )

        seen_predictions = {}
        nbest = []
        for pred in prelim_predictions:
            if len(nbest) >= n_best_size:
                break
            feature = features[pred.feature_index]

            # XLNet un-tokenizer
            # Let's keep it simple for now and see if we need all this later.
            #
            # tok_start_to_orig_index = feature.tok_start_to_orig_index
            # tok_end_to_orig_index = feature.tok_end_to_orig_index
            # start_orig_pos = tok_start_to_orig_index[pred.start_index]
            # end_orig_pos = tok_end_to_orig_index[pred.end_index]
            # paragraph_text = example.paragraph_text
            # final_text = paragraph_text[start_orig_pos: end_orig_pos + 1].strip()

            # Previously used Bert untokenizer
            tok_tokens = feature.tokens[pred.start_index : (pred.end_index + 1)]
            orig_doc_start = feature.token_to_orig_map[pred.start_index]
            orig_doc_end = feature.token_to_orig_map[pred.end_index]
            orig_tokens = example.doc_tokens[orig_doc_start : (orig_doc_end + 1)]
            tok_text = tokenizer.convert_tokens_to_string(tok_tokens)

            # Clean whitespace
            tok_text = tok_text.strip()
            tok_text = " ".join(tok_text.split())
            orig_text = " ".join(orig_tokens)

            if hasattr(tokenizer, "do_lower_case"):
                do_lower_case = tokenizer.do_lower_case
            else:
                do_lower_case = tokenizer.do_lowercase_and_remove_accent

            final_text = get_final_text(tok_text, orig_text, do_lower_case, verbose_logging)

            if final_text in seen_predictions:
                continue

            seen_predictions[final_text] = True

            nbest.append(
                _NbestPrediction(text=final_text, start_log_prob=pred.start_log_prob, end_log_prob=pred.end_log_prob)
            )

        # In very rare edge cases we could have no valid predictions. So we
        # just create a nonce prediction in this case to avoid failure.
        if not nbest:
            nbest.append(_NbestPrediction(text="", start_log_prob=-1e6, end_log_prob=-1e6))

        total_scores = []
        best_non_null_entry = None
        for entry in nbest:
            total_scores.append(entry.start_log_prob + entry.end_log_prob)
            if not best_non_null_entry:
                best_non_null_entry = entry

        probs = _compute_softmax(total_scores)

        nbest_json = []
        for i, entry in enumerate(nbest):
            output = collections.OrderedDict()
            output["text"] = entry.text
            output["probability"] = probs[i]
            output["start_log_prob"] = entry.start_log_prob
            output["end_log_prob"] = entry.end_log_prob
            nbest_json.append(output)

        if len(nbest_json) < 1:
            raise ValueError("No valid predictions")
        if best_non_null_entry is None:
            raise ValueError("No valid predictions")

        score_diff = score_null
        scores_diff_json[example.qas_id] = score_diff
        # note(zhiliny): always predict best_non_null_entry
        # and the evaluation script will search for the best threshold
        all_predictions[example.qas_id] = best_non_null_entry.text

        all_nbest_json[example.qas_id] = nbest_json

    with open(output_prediction_file, "w") as writer:
        writer.write(json.dumps(all_predictions, indent=4) + "\n")

    with open(output_nbest_file, "w") as writer:
        writer.write(json.dumps(all_nbest_json, indent=4) + "\n")

    if version_2_with_negative:
        with open(output_null_log_odds_file, "w") as writer:
            writer.write(json.dumps(scores_diff_json, indent=4) + "\n")



SUMMARY:
**Summary:**

The `compute_predictions_log_probs` function implements the prediction logic for XLNet, a language model. It calculates the predicted probabilities of text spans in a question given the provided input (questions and passages) and stores the outcomes in JSON files.

**Main Purpose and Functionality:**

The primary purpose is to generate predictions based on the input features and evaluate them against the ground truth.

**Key Implementation Details:**

*   Iterates over the input examples, features, results, and predicted text spans.
*   Calculates scores for all possible text spans given a start token position `i` (`start_n_top`) and an end token position `j` (`end_n_top`).
*   Sorts the predictions by their combined score in descending order.
*   Filters out invalid predictions based on conditions like valid answers and context limits.

**Notable Dependencies or Requirements:**

*   The `tokenizer` object, which is used to convert text tokens to strings and vice versa.
*   External dependency on the `utils_squad_evaluate.py` module for evaluating sequence evaluation metrics. 

**Additional Insights :**

*   This function assumes a knowledge of the Hugging Face library's tokenizer and its features (e.g., token IDs, original text positions).
*   The function provides utility functions and structures to assist with sequence prediction tasks.
*   Error handling has been implemented to ensure no valid predictions are returned when needed.


key: chroma:document
string_value:
CODE:

def _improve_answer_span(doc_tokens, input_start, input_end, tokenizer, orig_answer_text):
    """Returns tokenized answer spans that better match the annotated answer."""
    tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text))

    for new_start in range(input_start, input_end + 1):
        for new_end in range(input_end, new_start - 1, -1):
            text_span = " ".join(doc_tokens[new_start : (new_end + 1)])
            if text_span == tok_answer_text:
                return (new_start, new_end)



SUMMARY:
**Summary:**

The `_improve_answer_span` function optimizes the match between annotated answer spans and actual answer texts in a document. It iteratively checks for substring matches within the specified `input_start` to `input_end` range and returns the smallest matching span.

**Key Details:**

* Returns tokenized answer spans with updated start/end indices (`(new_start, new_end)`).
* Uses `tokenize()` from the tokenizer service to generate the original answer text (`tok_answer_text`).
* Implements a brute-force matching strategy with nested loops for checking substring matches within the document.

**Notable Dependencies:**

* `tokenizer`: an instance of a tokenization service (e.g., NLTK, spaCy).
* `doc_tokens`: a list or array of document tokens.
* `orig_answer_text`: the original answer text to be matched.


key: chroma:document
string_value:
CODE:

def _check_is_max_context(doc_spans, cur_span_index, position):
    """Check if this is the 'max context' doc span for the token."""
    best_score = None
    best_span_index = None
    for span_index, doc_span in enumerate(doc_spans):
        end = doc_span.start + doc_span.length - 1
        if position < doc_span.start:
            continue
        if position > end:
            continue
        num_left_context = position - doc_span.start
        num_right_context = end - position
        score = min(num_left_context, num_right_context) + 0.01 * doc_span.length
        if best_score is None or score > best_score:
            best_score = score
            best_span_index = span_index



SUMMARY:
**Function Summary: `_check_is_max_context`**

**Purpose**: Checks if the current token's position is within the 'max context' span in the document.

* primary purpose: Determine optimal contextual span for a token based on its position.

**Key Implementation Details**:

* Calculates a score for each possible span based on proximity to the token's position (`score = min(num_left_context, num_right_context) + 0.01 * doc_span.length`).
* Compares scores and selects the best span index when `best_score` is unavailable or surpassed.

**Notable Dependencies/Requirements**:

* `doc_spans`: list of spans in the document (accessed using `enumerate`)
* `position`: current token's position

No external dependencies mentioned.


key: chroma:document
string_value:
CODE:

def _new_check_is_max_context(doc_spans, cur_span_index, position):
    """Check if this is the 'max context' doc span for the token."""
    # if len(doc_spans) == 1:
    # return True
    best_score = None
    best_span_index = None
    for span_index, doc_span in enumerate(doc_spans):
        end = doc_span["start"] + doc_span["length"] - 1
        if position < doc_span["start"]:
            continue
        if position > end:
            continue
        num_left_context = position - doc_span["start"]
        num_right_context = end - position
        score = min(num_left_context, num_right_context) + 0.01 * doc_span["length"]
        if best_score is None or score > best_score:
            best_score = score
            best_span_index = span_index



SUMMARY:
**Summary: `_new_check_is_max_context` Function**

*   **Purpose:** Checks if the given `position` is within the 'max context' document span for a token, returning `True` if so.

*   **Key implementation details:**

    *   Calculates scores for left and right context based on distance from the position.
    *   Minimizes sum of left/context and right/context distances with a length bonus.
    *   Compares score against current best and updates it as necessary.
*   **Notable dependencies and requirements:**

    *   `doc_spans`: A list of document spans, where each span has "start" and "length" attributes.
    *   **Interfaces/Implementations:** None noted.

Note that the commented-out line suggests a trivial case handling for single-element doc_span lists.


key: chroma:document
string_value:
CODE:

def _is_whitespace(c):
    if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F:
        return True


SUMMARY:
**Summary**

This function, `_is_whitespace`, determines whether a given Unicode character `c` is classified as whitespace. It checks for the following conditions:

* Space (`" "`), Tab (`"\t"`), Line Feed (`"\n"`), and Vertical Tab (`"\r"`),
* The Unicode code point 0x202F (U+203FF, EN QUAD).

The function returns `True` if the condition is met, and `False` otherwise.


key: chroma:document
string_value:
CODE:

def squad_convert_example_to_features(
    example, max_seq_length, doc_stride, max_query_length, padding_strategy, is_training
):
    features = []
    if is_training and not example.is_impossible:
        # Get start and end position
        start_position = example.start_position
        end_position = example.end_position

        # If the answer cannot be found in the text, then skip this example.
        actual_text = " ".join(example.doc_tokens[start_position : (end_position + 1)])
        cleaned_answer_text = " ".join(whitespace_tokenize(example.answer_text))
        if actual_text.find(cleaned_answer_text) == -1:
            logger.warning(f"Could not find answer: '{actual_text}' vs. '{cleaned_answer_text}'")
            return []

    tok_to_orig_index = []
    orig_to_tok_index = []
    all_doc_tokens = []
    for i, token in enumerate(example.doc_tokens):
        orig_to_tok_index.append(len(all_doc_tokens))
        if tokenizer.__class__.__name__ in [
            "RobertaTokenizer",
            "LongformerTokenizer",
            "BartTokenizer",
            "RobertaTokenizerFast",
            "LongformerTokenizerFast",
            "BartTokenizerFast",
        ]:
            sub_tokens = tokenizer.tokenize(token, add_prefix_space=True)
        else:
            sub_tokens = tokenizer.tokenize(token)
        for sub_token in sub_tokens:
            tok_to_orig_index.append(i)
            all_doc_tokens.append(sub_token)

    if is_training and not example.is_impossible:
        tok_start_position = orig_to_tok_index[example.start_position]
        if example.end_position < len(example.doc_tokens) - 1:
            tok_end_position = orig_to_tok_index[example.end_position + 1] - 1
        else:
            tok_end_position = len(all_doc_tokens) - 1

        (tok_start_position, tok_end_position) = _improve_answer_span(
            all_doc_tokens, tok_start_position, tok_end_position, tokenizer, example.answer_text
        )

    spans = []

    truncated_query = tokenizer.encode(
        example.question_text, add_special_tokens=False, truncation=True, max_length=max_query_length
    )

    # Tokenizers who insert 2 SEP tokens in-between <context> & <question> need to have special handling
    # in the way they compute mask of added tokens.
    tokenizer_type = type(tokenizer).__name__.replace("Tokenizer", "").lower()
    sequence_added_tokens = (
        tokenizer.model_max_length - tokenizer.max_len_single_sentence + 1
        if tokenizer_type in MULTI_SEP_TOKENS_TOKENIZERS_SET
        else tokenizer.model_max_length - tokenizer.max_len_single_sentence
    )
    sequence_pair_added_tokens = tokenizer.model_max_length - tokenizer.max_len_sentences_pair

    span_doc_tokens = all_doc_tokens
    while len(spans) * doc_stride < len(all_doc_tokens):
        # Define the side we want to truncate / pad and the text/pair sorting
        if tokenizer.padding_side == "right":
            texts = truncated_query
            pairs = span_doc_tokens
            truncation = TruncationStrategy.ONLY_SECOND.value
        else:
            texts = span_doc_tokens
            pairs = truncated_query
            truncation = TruncationStrategy.ONLY_FIRST.value

        encoded_dict = tokenizer.encode_plus(  # TODO(thom) update this logic
            texts,
            pairs,
            truncation=truncation,
            padding=padding_strategy,
            max_length=max_seq_length,
            return_overflowing_tokens=True,
            stride=max_seq_length - doc_stride - len(truncated_query) - sequence_pair_added_tokens,
            return_token_type_ids=True,
        )

        paragraph_len = min(
            len(all_doc_tokens) - len(spans) * doc_stride,
            max_seq_length - len(truncated_query) - sequence_pair_added_tokens,
        )

        if tokenizer.pad_token_id in encoded_dict["input_ids"]:
            if tokenizer.padding_side == "right":
                non_padded_ids = encoded_dict["input_ids"][: encoded_dict["input_ids"].index(tokenizer.pad_token_id)]
            else:
                last_padding_id_position = (
                    len(encoded_dict["input_ids"]) - 1 - encoded_dict["input_ids"][::-1].index(tokenizer.pad_token_id)
                )
                non_padded_ids = encoded_dict["input_ids"][last_padding_id_position + 1 :]

        else:
            non_padded_ids = encoded_dict["input_ids"]

        tokens = tokenizer.convert_ids_to_tokens(non_padded_ids)

        token_to_orig_map = {}
        for i in range(paragraph_len):
            index = len(truncated_query) + sequence_added_tokens + i if tokenizer.padding_side == "right" else i
            token_to_orig_map[index] = tok_to_orig_index[len(spans) * doc_stride + i]

        encoded_dict["paragraph_len"] = paragraph_len
        encoded_dict["tokens"] = tokens
        encoded_dict["token_to_orig_map"] = token_to_orig_map
        encoded_dict["truncated_query_with_special_tokens_length"] = len(truncated_query) + sequence_added_tokens
        encoded_dict["token_is_max_context"] = {}
        encoded_dict["start"] = len(spans) * doc_stride
        encoded_dict["length"] = paragraph_len

        spans.append(encoded_dict)

        if "overflowing_tokens" not in encoded_dict or (
            "overflowing_tokens" in encoded_dict and len(encoded_dict["overflowing_tokens"]) == 0
        ):
            break
        span_doc_tokens = encoded_dict["overflowing_tokens"]

    for doc_span_index in range(len(spans)):
        for j in range(spans[doc_span_index]["paragraph_len"]):
            is_max_context = _new_check_is_max_context(spans, doc_span_index, doc_span_index * doc_stride + j)
            index = (
                j
                if tokenizer.padding_side == "left"
                else spans[doc_span_index]["truncated_query_with_special_tokens_length"] + j
            )
            spans[doc_span_index]["token_is_max_context"][index] = is_max_context

    for span in spans:
        # Identify the position of the CLS token
        cls_index = span["input_ids"].index(tokenizer.cls_token_id)

        # p_mask: mask with 1 for token than cannot be in the answer (0 for token which can be in an answer)
        # Original TF implementation also keep the classification token (set to 0)
        p_mask = np.ones_like(span["token_type_ids"])
        if tokenizer.padding_side == "right":
            p_mask[len(truncated_query) + sequence_added_tokens :] = 0
        else:
            p_mask[-len(span["tokens"]) : -(len(truncated_query) + sequence_added_tokens)] = 0

        pad_token_indices = np.where(span["input_ids"] == tokenizer.pad_token_id)
        special_token_indices = np.asarray(
            tokenizer.get_special_tokens_mask(span["input_ids"], already_has_special_tokens=True)
        ).nonzero()

        p_mask[pad_token_indices] = 1
        p_mask[special_token_indices] = 1

        # Set the cls index to 0: the CLS index can be used for impossible answers
        p_mask[cls_index] = 0

        span_is_impossible = example.is_impossible
        start_position = 0
        end_position = 0
        if is_training and not span_is_impossible:
            # For training, if our document chunk does not contain an annotation
            # we throw it out, since there is nothing to predict.
            doc_start = span["start"]
            doc_end = span["start"] + span["length"] - 1
            out_of_span = False

            if not (tok_start_position >= doc_start and tok_end_position <= doc_end):
                out_of_span = True

            if out_of_span:
                start_position = cls_index
                end_position = cls_index
                span_is_impossible = True
            else:
                if tokenizer.padding_side == "left":
                    doc_offset = 0
                else:
                    doc_offset = len(truncated_query) + sequence_added_tokens

                start_position = tok_start_position - doc_start + doc_offset
                end_position = tok_end_position - doc_start + doc_offset

        features.append(
            SquadFeatures(
                span["input_ids"],
                span["attention_mask"],
                span["token_type_ids"],
                cls_index,
                p_mask.tolist(),
                example_index=0,  # Can not set unique_id and example_index here. They will be set after multiple processing.
                unique_id=0,
                paragraph_len=span["paragraph_len"],
                token_is_max_context=span["token_is_max_context"],
                tokens=span["tokens"],
                token_to_orig_map=span["token_to_orig_map"],
                start_position=start_position,
                end_position=end_position,
                is_impossible=span_is_impossible,
                qas_id=example.qas_id,
            )
        )


SUMMARY:
This function, `squad_convert_example_to_features`, converts a Natural Language Processing (NLP) example into a set of features suitable for training a question-answering model. The input "example" is expected to have various attributes such as `doc_tokens`, `answer_text`, `question_text`, and `is_impossible`.

The function performs the following steps:

1. Extracts relevant information from the input example, including start and end positions.
2. Processes the text data using a specified tokenizer (such as Hugging Face's Tokenizers) to produce encoded representations of the context and query texts.
3. Truncates the query text based on the length of the truncated query with special tokens to avoid excessive padding.
4. Identifies span boundaries within the processed documents based on the start and end positions.
5. Generates unique features for each span, including input IDs, attention masks, token type IDs, classification masks, and other metadata.

The function returns a list of `SquadFeatures` objects, which represent individual spans in the input example. These features can be used as input data for training a question-answering model, such as BERT.

Overall, this function is a critical component in converting raw NLP examples into a format suitable for training and evaluating question-answering models.


key: chroma:document
string_value:
CODE:

def squad_convert_example_to_features_init(tokenizer_for_convert: PreTrainedTokenizerBase):
    global tokenizer


SUMMARY:
**Function Summary:**

`def squad_convert_example_to_features_init(tokenizer_for_convert: PreTrainedTokenizerBase)`

**Main Purpose and Functionality:**
Initializes the Global Tokenizer with a Procedurally Converted Model's Tokenizer.

**Key Implementation Details:**

- Reinitializes the `tokenizer` variable with the provided `tokenizer_for_convert`.
- Uses the `PreTrainedTokenizerBase` type hint to ensure compatibility.
  
**Notable Dependencies or Requirements:**

- `tokenizer`: Global variable, reassigned with `tokenizer_for_convert`.
- External dependency: Hugging Face Transformers library (`PreTrainedTokenizerBase`).


key: chroma:document
string_value:
CODE:

def squad_convert_examples_to_features(
    examples,
    tokenizer,
    max_seq_length,
    doc_stride,
    max_query_length,
    is_training,
    padding_strategy="max_length",
    return_dataset=False,
    threads=1,
    tqdm_enabled=True,
):
    """
    Converts a list of examples into a list of features that can be directly given as input to a model. It is
    model-dependant and takes advantage of many of the tokenizer's features to create the model's inputs.

    Args:
        examples: list of [`~data.processors.squad.SquadExample`]
        tokenizer: an instance of a child of [`PreTrainedTokenizer`]
        max_seq_length: The maximum sequence length of the inputs.
        doc_stride: The stride used when the context is too large and is split across several features.
        max_query_length: The maximum length of the query.
        is_training: whether to create features for model evaluation or model training.
        padding_strategy: Default to "max_length". Which padding strategy to use
        return_dataset: Default False. Either 'pt' or 'tf'.
            if 'pt': returns a torch.data.TensorDataset, if 'tf': returns a tf.data.Dataset
        threads: multiple processing threads.


    Returns:
        list of [`~data.processors.squad.SquadFeatures`]

    Example:

    ```python
    processor = SquadV2Processor()
    examples = processor.get_dev_examples(data_dir)

    features = squad_convert_examples_to_features(
        examples=examples,
        tokenizer=tokenizer,
        max_seq_length=args.max_seq_length,
        doc_stride=args.doc_stride,
        max_query_length=args.max_query_length,
        is_training=not evaluate,
    )
    ```"""
    # Defining helper methods
    features = []

    threads = min(threads, cpu_count())
    with Pool(threads, initializer=squad_convert_example_to_features_init, initargs=(tokenizer,)) as p:
        annotate_ = partial(
            squad_convert_example_to_features,
            max_seq_length=max_seq_length,
            doc_stride=doc_stride,
            max_query_length=max_query_length,
            padding_strategy=padding_strategy,
            is_training=is_training,
        )
        features = list(
            tqdm(
                p.imap(annotate_, examples, chunksize=32),
                total=len(examples),
                desc="convert squad examples to features",
                disable=not tqdm_enabled,
            )
        )

    new_features = []
    unique_id = 1000000000
    example_index = 0
    for example_features in tqdm(
        features, total=len(features), desc="add example index and unique id", disable=not tqdm_enabled
    ):
        if not example_features:
            continue
        for example_feature in example_features:
            example_feature.example_index = example_index
            example_feature.unique_id = unique_id
            new_features.append(example_feature)
            unique_id += 1
        example_index += 1
    features = new_features
    del new_features
    if return_dataset == "pt":
        if not is_torch_available():
            raise RuntimeError("PyTorch must be installed to return a PyTorch dataset.")

        # Convert to Tensors and build dataset
        all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
        all_attention_masks = torch.tensor([f.attention_mask for f in features], dtype=torch.long)
        all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
        all_cls_index = torch.tensor([f.cls_index for f in features], dtype=torch.long)
        all_p_mask = torch.tensor([f.p_mask for f in features], dtype=torch.float)
        all_is_impossible = torch.tensor([f.is_impossible for f in features], dtype=torch.float)

        if not is_training:
            all_feature_index = torch.arange(all_input_ids.size(0), dtype=torch.long)
            dataset = TensorDataset(
                all_input_ids, all_attention_masks, all_token_type_ids, all_feature_index, all_cls_index, all_p_mask
            )
        else:
            all_start_positions = torch.tensor([f.start_position for f in features], dtype=torch.long)
            all_end_positions = torch.tensor([f.end_position for f in features], dtype=torch.long)
            dataset = TensorDataset(
                all_input_ids,
                all_attention_masks,
                all_token_type_ids,
                all_start_positions,
                all_end_positions,
                all_cls_index,
                all_p_mask,
                all_is_impossible,
            )

        return features, dataset
    elif return_dataset == "tf":
        if not is_tf_available():
            raise RuntimeError("TensorFlow must be installed to return a TensorFlow dataset.")

        def gen():
            for i, ex in enumerate(features):
                if ex.token_type_ids is None:
                    yield (
                        {
                            "input_ids": ex.input_ids,
                            "attention_mask": ex.attention_mask,
                            "feature_index": i,
                            "qas_id": ex.qas_id,
                        },
                        {
                            "start_positions": ex.start_position,
                            "end_positions": ex.end_position,
                            "cls_index": ex.cls_index,
                            "p_mask": ex.p_mask,
                            "is_impossible": ex.is_impossible,
                        },
                    )
                else:
                    yield (
                        {
                            "input_ids": ex.input_ids,
                            "attention_mask": ex.attention_mask,
                            "token_type_ids": ex.token_type_ids,
                            "feature_index": i,
                            "qas_id": ex.qas_id,
                        },
                        {
                            "start_positions": ex.start_position,
                            "end_positions": ex.end_position,
                            "cls_index": ex.cls_index,
                            "p_mask": ex.p_mask,
                            "is_impossible": ex.is_impossible,
                        },
                    )

        # Why have we split the batch into a tuple? PyTorch just has a list of tensors.
        if "token_type_ids" in tokenizer.model_input_names:
            train_types = (
                {
                    "input_ids": tf.int32,
                    "attention_mask": tf.int32,
                    "token_type_ids": tf.int32,
                    "feature_index": tf.int64,
                    "qas_id": tf.string,
                },
                {
                    "start_positions": tf.int64,
                    "end_positions": tf.int64,
                    "cls_index": tf.int64,
                    "p_mask": tf.int32,
                    "is_impossible": tf.int32,
                },
            )

            train_shapes = (
                {
                    "input_ids": tf.TensorShape([None]),
                    "attention_mask": tf.TensorShape([None]),
                    "token_type_ids": tf.TensorShape([None]),
                    "feature_index": tf.TensorShape([]),
                    "qas_id": tf.TensorShape([]),
                },
                {
                    "start_positions": tf.TensorShape([]),
                    "end_positions": tf.TensorShape([]),
                    "cls_index": tf.TensorShape([]),
                    "p_mask": tf.TensorShape([None]),
                    "is_impossible": tf.TensorShape([]),
                },
            )
        else:
            train_types = (
                {"input_ids": tf.int32, "attention_mask": tf.int32, "feature_index": tf.int64, "qas_id": tf.string},
                {
                    "start_positions": tf.int64,
                    "end_positions": tf.int64,
                    "cls_index": tf.int64,
                    "p_mask": tf.int32,
                    "is_impossible": tf.int32,
                },
            )

            train_shapes = (
                {
                    "input_ids": tf.TensorShape([None]),
                    "attention_mask": tf.TensorShape([None]),
                    "feature_index": tf.TensorShape([]),
                    "qas_id": tf.TensorShape([]),
                },
                {
                    "start_positions": tf.TensorShape([]),
                    "end_positions": tf.TensorShape([]),
                    "cls_index": tf.TensorShape([]),
                    "p_mask": tf.TensorShape([None]),
                    "is_impossible": tf.TensorShape([]),
                },
            )

        return tf.data.Dataset.from_generator(gen, train_types, train_shapes)
    else:


SUMMARY:
**Function Summary**

` squad_convert_examples_to_features`

Converts a list of `SquadExample` objects into a list of `SquadFeatures` that can be directly inputted to a model.

**Functionsality**

* Takes in a list of `SquadExample` objects, tokenizer, and various parameters (max_seq_length, doc_stride, max_query_length, etc.)
* Converts the examples into features using the tokenizer
* Optionally returns a PyTorch or TensorFlow dataset based on the input parameter

**Key Implementation Details**

* Utilizes parallel processing through PyTorch's `Pool` class to convert each example to features in parallel
* Uses `tqdm` library for progress bars
* Converts examples to either PyTorch or TensorFlow datasets based on the input parameter

**Dependent Variables and Interfaces**

* tokenizer (an instance of a model tokenizer)
* SquadExample objects
* SquadFeatures objects
* PyTorch or TensorFlow datasets

**Notes**

* The function returns a list of features that can be directly inputted to a model
* The function also returns a dataset object based on the input parameter, which can be used for training a machine learning model.


key: chroma:document
string_value:
CODE:

class SquadProcessor(DataProcessor):
    """
    Processor for the SQuAD data set. overridden by SquadV1Processor and SquadV2Processor, used by the version 1.1 and
    version 2.0 of SQuAD, respectively.
    """

    train_file = None
    dev_file = None

    def _get_example_from_tensor_dict(self, tensor_dict, evaluate=False):
        if not evaluate:
            answer = tensor_dict["answers"]["text"][0].numpy().decode("utf-8")
            answer_start = tensor_dict["answers"]["answer_start"][0].numpy()
            answers = []
        else:
            answers = [
                {"answer_start": start.numpy(), "text": text.numpy().decode("utf-8")}
                for start, text in zip(tensor_dict["answers"]["answer_start"], tensor_dict["answers"]["text"])
            ]

            answer = None
            answer_start = None

        return SquadExample(
            qas_id=tensor_dict["id"].numpy().decode("utf-8"),
            question_text=tensor_dict["question"].numpy().decode("utf-8"),
            context_text=tensor_dict["context"].numpy().decode("utf-8"),
            answer_text=answer,
            start_position_character=answer_start,
            title=tensor_dict["title"].numpy().decode("utf-8"),
            answers=answers,
        )

    def get_examples_from_dataset(self, dataset, evaluate=False):
        """
        Creates a list of [`~data.processors.squad.SquadExample`] using a TFDS dataset.

        Args:
            dataset: The tfds dataset loaded from *tensorflow_datasets.load("squad")*
            evaluate: Boolean specifying if in evaluation mode or in training mode

        Returns:
            List of SquadExample

        Examples:

        ```python
        >>> import tensorflow_datasets as tfds

        >>> dataset = tfds.load("squad")

        >>> training_examples = get_examples_from_dataset(dataset, evaluate=False)
        >>> evaluation_examples = get_examples_from_dataset(dataset, evaluate=True)
        ```"""

        if evaluate:
            dataset = dataset["validation"]
        else:
            dataset = dataset["train"]

        examples = []
        for tensor_dict in tqdm(dataset):
            examples.append(self._get_example_from_tensor_dict(tensor_dict, evaluate=evaluate))

        return examples

    def get_train_examples(self, data_dir, filename=None):
        """
        Returns the training examples from the data directory.

        Args:
            data_dir: Directory containing the data files used for training and evaluating.
            filename: None by default, specify this if the training file has a different name than the original one
                which is `train-v1.1.json` and `train-v2.0.json` for squad versions 1.1 and 2.0 respectively.

        """
        if data_dir is None:
            data_dir = ""

        if self.train_file is None:
            raise ValueError("SquadProcessor should be instantiated via SquadV1Processor or SquadV2Processor")

        with open(
            os.path.join(data_dir, self.train_file if filename is None else filename), "r", encoding="utf-8"
        ) as reader:
            input_data = json.load(reader)["data"]
        return self._create_examples(input_data, "train")

    def get_dev_examples(self, data_dir, filename=None):
        """
        Returns the evaluation example from the data directory.

        Args:
            data_dir: Directory containing the data files used for training and evaluating.
            filename: None by default, specify this if the evaluation file has a different name than the original one
                which is `dev-v1.1.json` and `dev-v2.0.json` for squad versions 1.1 and 2.0 respectively.
        """
        if data_dir is None:
            data_dir = ""

        if self.dev_file is None:
            raise ValueError("SquadProcessor should be instantiated via SquadV1Processor or SquadV2Processor")

        with open(
            os.path.join(data_dir, self.dev_file if filename is None else filename), "r", encoding="utf-8"
        ) as reader:
            input_data = json.load(reader)["data"]
        return self._create_examples(input_data, "dev")

    def _create_examples(self, input_data, set_type):
        is_training = set_type == "train"
        examples = []
        for entry in tqdm(input_data):
            title = entry["title"]
            for paragraph in entry["paragraphs"]:
                context_text = paragraph["context"]
                for qa in paragraph["qas"]:
                    qas_id = qa["id"]
                    question_text = qa["question"]
                    start_position_character = None
                    answer_text = None
                    answers = []

                    is_impossible = qa.get("is_impossible", False)
                    if not is_impossible:
                        if is_training:
                            answer = qa["answers"][0]
                            answer_text = answer["text"]
                            start_position_character = answer["answer_start"]
                        else:
                            answers = qa["answers"]

                    example = SquadExample(
                        qas_id=qas_id,
                        question_text=question_text,
                        context_text=context_text,
                        answer_text=answer_text,
                        start_position_character=start_position_character,
                        title=title,
                        is_impossible=is_impossible,
                        answers=answers,
                    )
                    examples.append(example)


SUMMARY:
**SquadProcessor Class Summary**

### Purpose and Functionality

The SquadProcessor class is designed to process SQuAD data, a dataset of natural language inference tasks. It provides methods for loading training and development datasets, as well as creating examples from these datasets.

### Implementation Details

* The `_get_example_from_tensor_dict` method creates a `SquadExample` object from a TFDS tensor dictionary.
* The `get_examples_from_dataset` method loads examples from a TFDS dataset in either evaluation or training mode.
* The `get_train_examples` and `get_dev_examples` methods load examples from files with specific naming conventions.
* The `_create_examples` method generates examples from input data, including title, paragraph context, questions, and answers.

### Notable Dependencies/Requirements

* `tensorflow_datasets` (TFDS) for loading SQuAD datasets
* `json` module for parsing JSON files
* `SquadExample` class (part of this codebase)

### Key Algorithm/Pattern

The SquadProcessor uses TFDS to load datasets and extract examples. It relies on the `_get_example_from_tensor_dict` method to convert tensor dictionaries into `SquadExample` objects.

### Important Method Signatures and Interfaces

* `get_examples_from_dataset(dataset, evaluate=False)`:
	+ Returns: List of `SquadExample`
	+ Args: `dataset`, `evaluate`

```python
class SquadProcessor(DataProcessor):
    # ...
    
    def get_examples_from_dataset(self, dataset, evaluate=False):
        """Creates a list of SquadExample instances."""
        
        if evaluate:
            dataset = dataset["validation"]
        else:
            dataset = dataset["train"]

        examples = []
        for tensor_dict in tqdm(dataset):
            examples.append(self._get_example_from_tensor_dict(tensor_dict, evaluate=evaluate))

        return examples
```

```python
def get_train_examples(self, data_dir, filename=None):
    """Returns the training examples from the data directory."""
    
    # ...
    
def get_dev_examples(self, data_dir, filename=None):
    """Returns the evaluation example from the data directory."""
    
    # ...
```


key: chroma:document
string_value:
CODE:

class SquadV1Processor(SquadProcessor):
    train_file = "train-v1.1.json"


SUMMARY:
**Summary:**

The `SquadV1Processor` class inherits from `SquadProcessor`. It defines two file-specific attributes:

* `train_file`: `"train-v1.1.json"`
* `dev_file`: `"dev-v1.1.json"`

These files appear to be input data for processing, with their specific versions (v1.1) hinting at a particular dataset or benchmark.

**Key Implementation Details:**

* No specific implementation details are visible in this snippet, suggesting that this class primarily serves as an identifier or configurer for the `SquadProcessor` instance.
* No algorithms, patterns, or design choices are explicit here; however, the fact that it's named `SquadV1Processor` implies a relation to the Stanford Question Answering Dataset (SQuAD) and its v1.1 version.

**Notable Dependencies or Requirements:**

* The class inherits from `SquadProcessor`, which is not shown in this snippet. However, it suggests an external dependency on the `SquadProcessor` implementation.
* External dependencies:
	+ None explicitly stated here; however, using these files (`train-v1.1.json` and `dev-v1.1.json`) implies a requirement for accessing external data or files.


key: chroma:document
string_value:
CODE:

class SquadV2Processor(SquadProcessor):
    train_file = "train-v2.0.json"


SUMMARY:
**Concise Summary**

The `SquadV2Processor` class inherits from `SquadProcessor`. It introduces the following key parameters:

* `train_file` and `dev_file`, referencing external JSON files, indicating specific input datasets for training and evaluation.

**Key Implementation Contexts:**

This processor is part of a natural language processing (NLP) context, specifically designed to process data related to the Stanford Question Answering Dataset v2.0.

**Notable Dependencies/Requirements:**

* `SquadProcessor` class
* `train-v2.0.json` and `dev-v2.0.json` external data files


key: chroma:document
string_value:
CODE:

class SquadExample:
    """
    A single training/test example for the Squad dataset, as loaded from disk.

    Args:
        qas_id: The example's unique identifier
        question_text: The question string
        context_text: The context string
        answer_text: The answer string
        start_position_character: The character position of the start of the answer
        title: The title of the example
        answers: None by default, this is used during evaluation. Holds answers as well as their start positions.
        is_impossible: False by default, set to True if the example has no possible answer.
    """

    def __init__(
        self,
        qas_id,
        question_text,
        context_text,
        answer_text,
        start_position_character,
        title,
        answers=[],
        is_impossible=False,
    ):
        self.qas_id = qas_id
        self.question_text = question_text
        self.context_text = context_text
        self.answer_text = answer_text
        self.title = title
        self.is_impossible = is_impossible
        self.answers = answers

        self.start_position, self.end_position = 0, 0

        doc_tokens = []
        char_to_word_offset = []
        prev_is_whitespace = True

        # Split on whitespace so that different tokens may be attributed to their original position.
        for c in self.context_text:
            if _is_whitespace(c):
                prev_is_whitespace = True
            else:
                if prev_is_whitespace:
                    doc_tokens.append(c)
                else:
                    doc_tokens[-1] += c
                prev_is_whitespace = False
            char_to_word_offset.append(len(doc_tokens) - 1)

        self.doc_tokens = doc_tokens
        self.char_to_word_offset = char_to_word_offset

        # Start and end positions only has a value during evaluation.
        if start_position_character is not None and not is_impossible:
            self.start_position = char_to_word_offset[start_position_character]
            self.end_position = char_to_word_offset[
                min(start_position_character + len(answer_text) - 1, len(char_to_word_offset) - 1)


SUMMARY:
**Summary:**

The `SquadExample` class represents a single training/test example for the Squad dataset. It encapsulates various attributes and methods related to text examples.

**Main Purpose and Functionality:**

- Initializes an example with its unique identifier (`qas_id`), question-text, context-text, answer-text, start-position-character, title, answers, and is-impossible flag.
  
- Splits the context-text into tokens and stores them in `doc_tokens`.
  
- Stores character-to-word offsets for each token.
  
**Key Implementation Details:**

- Tokenization of context text at the character level using a loop.

- Calculation of start_position and end_position when provided input (non-impossible example).

**Notable Dependencies or Requirements:**

- `_is_whitespace(c)` function not defined in this snippet, assumed as part of external dependencies.
 
- External dependencies:
  - `_is_whitespace(c)`: a whitespace check function
  

 **Interfaces Implemented:**

Not applicable.


key: chroma:document
string_value:
CODE:

class SquadFeatures:
    """
    Single squad example features to be fed to a model. Those features are model-specific and can be crafted from
    [`~data.processors.squad.SquadExample`] using the
    :method:*~transformers.data.processors.squad.squad_convert_examples_to_features* method.

    Args:
        input_ids: Indices of input sequence tokens in the vocabulary.
        attention_mask: Mask to avoid performing attention on padding token indices.
        token_type_ids: Segment token indices to indicate first and second portions of the inputs.
        cls_index: the index of the CLS token.
        p_mask: Mask identifying tokens that can be answers vs. tokens that cannot.
            Mask with 1 for tokens than cannot be in the answer and 0 for token that can be in an answer
        example_index: the index of the example
        unique_id: The unique Feature identifier
        paragraph_len: The length of the context
        token_is_max_context:
            List of booleans identifying which tokens have their maximum context in this feature object. If a token
            does not have their maximum context in this feature object, it means that another feature object has more
            information related to that token and should be prioritized over this feature for that token.
        tokens: list of tokens corresponding to the input ids
        token_to_orig_map: mapping between the tokens and the original text, needed in order to identify the answer.
        start_position: start of the answer token index
        end_position: end of the answer token index
        encoding: optionally store the BatchEncoding with the fast-tokenizer alignment methods.
    """

    def __init__(
        self,
        input_ids,
        attention_mask,
        token_type_ids,
        cls_index,
        p_mask,
        example_index,
        unique_id,
        paragraph_len,
        token_is_max_context,
        tokens,
        token_to_orig_map,
        start_position,
        end_position,
        is_impossible,
        qas_id: str = None,
        encoding: BatchEncoding = None,
    ):
        self.input_ids = input_ids
        self.attention_mask = attention_mask
        self.token_type_ids = token_type_ids
        self.cls_index = cls_index
        self.p_mask = p_mask

        self.example_index = example_index
        self.unique_id = unique_id
        self.paragraph_len = paragraph_len
        self.token_is_max_context = token_is_max_context
        self.tokens = tokens
        self.token_to_orig_map = token_to_orig_map

        self.start_position = start_position
        self.end_position = end_position
        self.is_impossible = is_impossible
        self.qas_id = qas_id



SUMMARY:
**Class Summary: SquadFeatures**

The `SquadFeatures` class represents a single squad example, containing essential data for model training. It captures the characteristics of an input sequence, including:

* Input sequence tokens (`input_ids`)
* Token types and attention mask (`attention_mask`, `token_type_ids`)
* Classification labels (`p_mask`, `cls_index`)
* Information on token mapping and positions in a paragraph (`paragraph_len`, `start_position`, `end_position`)
* Additional flags for identifying impossible tokens (`is_impossible`) and corresponding context (`token_is_max_context`)

**Key Implementation Details:**

* The class uses a detailed mapping between input tokens and original text (`token_to_orig_map`)
* It incorporates BatchEncoding data with fast-tokenizer alignment methods, if specified

**Notable Dependencies and Requirements:**

* Related classes/functons:
	+ `SquadExample`
	+ `transformers.data.processors.squad.squad_convert_examples_to_features`
	+ `BatchEncoding`
* External dependencies:
	+ `transformers` library for token processing
	+ `fast-tokenizer` library (if using Batch Encoding)
* Interfaces implemented/inherited:
	+ `__init__` with all necessary keyword arguments


key: chroma:document
string_value:
CODE:

class SquadResult:
    """
    Constructs a SquadResult which can be used to evaluate a model's output on the SQuAD dataset.

    Args:
        unique_id: The unique identifier corresponding to that example.
        start_logits: The logits corresponding to the start of the answer
        end_logits: The logits corresponding to the end of the answer
    """

    def __init__(self, unique_id, start_logits, end_logits, start_top_index=None, end_top_index=None, cls_logits=None):
        self.start_logits = start_logits
        self.end_logits = end_logits
        self.unique_id = unique_id

        if start_top_index:
            self.start_top_index = start_top_index
            self.end_top_index = end_top_index


SUMMARY:
**SquadResult Class Summary**

#### Main Purpose and Functionality
The SquadResult class represents an output evaluation result for the SQuAD dataset. It constructs a model's answer span indices, logits, and corresponding top indexes.

#### Key Implementation Details

* Initializes with unique ID, start and end logits.
* Optionally accepts top indexes (start_top_index, end_top_index) and class logits (cls_logits) to provide additional context.

#### Notable Dependencies or Requirements
None noted.


key: chroma:document
string_value:
CODE:

class XnliProcessor(DataProcessor):
    """
    Processor for the XNLI dataset. Adapted from
    https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/run_classifier.py#L207
    """

    def __init__(self, language, train_language=None):
        self.language = language
        self.train_language = train_language

    def get_train_examples(self, data_dir):
        """See base class."""
        lg = self.language if self.train_language is None else self.train_language
        lines = self._read_tsv(os.path.join(data_dir, f"XNLI-MT-1.0/multinli/multinli.train.{lg}.tsv"))
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"train-{i}"
            text_a = line[0]
            text_b = line[1]
            label = "contradiction" if line[2] == "contradictory" else line[2]
            if not isinstance(text_a, str):
                raise TypeError(f"Training input {text_a} is not a string")
            if not isinstance(text_b, str):
                raise TypeError(f"Training input {text_b} is not a string")
            if not isinstance(label, str):
                raise TypeError(f"Training label {label} is not a string")
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
        return examples

    def get_test_examples(self, data_dir):
        """See base class."""
        lines = self._read_tsv(os.path.join(data_dir, "XNLI-1.0/xnli.test.tsv"))
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            language = line[0]
            if language != self.language:
                continue
            guid = f"test-{i}"
            text_a = line[6]
            text_b = line[7]
            label = line[1]
            if not isinstance(text_a, str):
                raise TypeError(f"Training input {text_a} is not a string")
            if not isinstance(text_b, str):
                raise TypeError(f"Training input {text_b} is not a string")
            if not isinstance(label, str):
                raise TypeError(f"Training label {label} is not a string")
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
        return examples

    def get_labels(self):
        """See base class."""


SUMMARY:
**Class Summary: XnliProcessor**

The `XnliProcessor` class is a processor for the XNLI (eXtreme Natural Language Understanding) dataset, adapted from Google Research's BERT project. It processes multilingual data for contrasting pairs.

**Key Functionality:**

* **Initialization**: Constructor (`__init__`) takes `language` and optional `train_language` as inputs.
* **Data Loading**: `get_train_examples` and `get_test_examples` methods load training and testing examples from TSV files.
	+ Returns a list of `InputExample` objects with `guid`, `text_a`, `text_b`, and `label` attributes.

**Implementation Details:**

* `_read_tsv` method reads TSV file lines
* Input validation for `text_a`, `text_b`, and `label`
* Filtering examples based on language in `get_test_examples`

**Dependencies and Requirements:**

* `os` module for file path manipulation
* `InputExample` class from the parent class (`DataProcessor`)
* `tsv` file with structured data


key: chroma:document
string_value:
CODE:

class DataProcessor:
    """Base class for data converters for sequence classification data sets."""

    def get_example_from_tensor_dict(self, tensor_dict):
        """
        Gets an example from a dict with tensorflow tensors.

        Args:
            tensor_dict: Keys and values should match the corresponding Glue
                tensorflow_dataset examples.
        """
        raise NotImplementedError()

    def get_train_examples(self, data_dir):
        """Gets a collection of [`InputExample`] for the train set."""
        raise NotImplementedError()

    def get_dev_examples(self, data_dir):
        """Gets a collection of [`InputExample`] for the dev set."""
        raise NotImplementedError()

    def get_test_examples(self, data_dir):
        """Gets a collection of [`InputExample`] for the test set."""
        raise NotImplementedError()

    def get_labels(self):
        """Gets the list of labels for this data set."""
        raise NotImplementedError()

    def tfds_map(self, example):
        """
        Some tensorflow_datasets datasets are not formatted the same way the GLUE datasets are. This method converts
        examples to the correct format.
        """
        if len(self.get_labels()) > 1:
            example.label = self.get_labels()[int(example.label)]
        return example

    @classmethod
    def _read_tsv(cls, input_file, quotechar=None):
        """Reads a tab separated value file."""
        with open(input_file, "r", encoding="utf-8-sig") as f:


SUMMARY:
**Concise Summary**

The `DataProcessor` class is a base class for data converters in sequence classification datasets. It provides methods for:

* Getting examples from tensor dictionaries (`get_example_from_tensor_dict`)
* Retrieving training, development, and testing examples (`get_train_examples`, `get_dev_examples`, `get_test_examples`)
* Retrieving labels (`get_labels`)
* Converting examples to the correct format using TensorFlow Datasets (`tfds_map`)

**Key Implementation Details**

* The `_read_tsv` class method reads a tab-separated value file
	+ Args: `input_file`, `quotechar`
* Important method signatures:
	+ `get_example_from_tensor_dict(tensor_dict)`
	+ `get_labels()`

**Notable Dependencies or Requirements**

* TensorFlow and/or TensorFlow Datasets
* Glue library ( referenced by the `tensorflow_dataset` import)
* CSV reader from the standard library


key: chroma:document
string_value:
CODE:

class SingleSentenceClassificationProcessor(DataProcessor):
    """Generic processor for a single sentence classification data set."""

    def __init__(self, labels=None, examples=None, mode="classification", verbose=False):
        self.labels = [] if labels is None else labels
        self.examples = [] if examples is None else examples
        self.mode = mode
        self.verbose = verbose

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, idx):
        if isinstance(idx, slice):
            return SingleSentenceClassificationProcessor(labels=self.labels, examples=self.examples[idx])
        return self.examples[idx]

    @classmethod
    def create_from_csv(
        cls, file_name, split_name="", column_label=0, column_text=1, column_id=None, skip_first_row=False, **kwargs
    ):
        processor = cls(**kwargs)
        processor.add_examples_from_csv(
            file_name,
            split_name=split_name,
            column_label=column_label,
            column_text=column_text,
            column_id=column_id,
            skip_first_row=skip_first_row,
            overwrite_labels=True,
            overwrite_examples=True,
        )
        return processor

    @classmethod
    def create_from_examples(cls, texts_or_text_and_labels, labels=None, **kwargs):
        processor = cls(**kwargs)
        processor.add_examples(texts_or_text_and_labels, labels=labels)
        return processor

    def add_examples_from_csv(
        self,
        file_name,
        split_name="",
        column_label=0,
        column_text=1,
        column_id=None,
        skip_first_row=False,
        overwrite_labels=False,
        overwrite_examples=False,
    ):
        lines = self._read_tsv(file_name)
        if skip_first_row:
            lines = lines[1:]
        texts = []
        labels = []
        ids = []
        for i, line in enumerate(lines):
            texts.append(line[column_text])
            labels.append(line[column_label])
            if column_id is not None:
                ids.append(line[column_id])
            else:
                guid = f"{split_name}-{i}" if split_name else str(i)
                ids.append(guid)

        return self.add_examples(
            texts, labels, ids, overwrite_labels=overwrite_labels, overwrite_examples=overwrite_examples
        )

    def add_examples(
        self, texts_or_text_and_labels, labels=None, ids=None, overwrite_labels=False, overwrite_examples=False
    ):
        if labels is not None and len(texts_or_text_and_labels) != len(labels):
            raise ValueError(
                f"Text and labels have mismatched lengths {len(texts_or_text_and_labels)} and {len(labels)}"
            )
        if ids is not None and len(texts_or_text_and_labels) != len(ids):
            raise ValueError(f"Text and ids have mismatched lengths {len(texts_or_text_and_labels)} and {len(ids)}")
        if ids is None:
            ids = [None] * len(texts_or_text_and_labels)
        if labels is None:
            labels = [None] * len(texts_or_text_and_labels)
        examples = []
        added_labels = set()
        for text_or_text_and_label, label, guid in zip(texts_or_text_and_labels, labels, ids):
            if isinstance(text_or_text_and_label, (tuple, list)) and label is None:
                text, label = text_or_text_and_label
            else:
                text = text_or_text_and_label
            added_labels.add(label)
            examples.append(InputExample(guid=guid, text_a=text, text_b=None, label=label))

        # Update examples
        if overwrite_examples:
            self.examples = examples
        else:
            self.examples.extend(examples)

        # Update labels
        if overwrite_labels:
            self.labels = list(added_labels)
        else:
            self.labels = list(set(self.labels).union(added_labels))

        return self.examples

    def get_features(
        self,
        tokenizer,
        max_length=None,
        pad_on_left=False,
        pad_token=0,
        mask_padding_with_zero=True,
        return_tensors=None,
    ):
        """
        Convert examples in a list of `InputFeatures`

        Args:
            tokenizer: Instance of a tokenizer that will tokenize the examples
            max_length: Maximum example length
            pad_on_left: If set to `True`, the examples will be padded on the left rather than on the right (default)
            pad_token: Padding token
            mask_padding_with_zero: If set to `True`, the attention mask will be filled by `1` for actual values
                and by `0` for padded values. If set to `False`, inverts it (`1` for padded values, `0` for actual
                values)

        Returns:
            If the `examples` input is a `tf.data.Dataset`, will return a `tf.data.Dataset` containing the
            task-specific features. If the input is a list of `InputExamples`, will return a list of task-specific
            `InputFeatures` which can be fed to the model.

        """
        if max_length is None:
            max_length = tokenizer.max_len

        label_map = {label: i for i, label in enumerate(self.labels)}

        all_input_ids = []
        for ex_index, example in enumerate(self.examples):
            if ex_index % 10000 == 0:
                logger.info(f"Tokenizing example {ex_index}")

            input_ids = tokenizer.encode(
                example.text_a,
                add_special_tokens=True,
                max_length=min(max_length, tokenizer.max_len),
            )
            all_input_ids.append(input_ids)

        batch_length = max(len(input_ids) for input_ids in all_input_ids)

        features = []
        for ex_index, (input_ids, example) in enumerate(zip(all_input_ids, self.examples)):
            if ex_index % 10000 == 0:
                logger.info(f"Writing example {ex_index}/{len(self.examples)}")
            # The mask has 1 for real tokens and 0 for padding tokens. Only real
            # tokens are attended to.
            attention_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)

            # Zero-pad up to the sequence length.
            padding_length = batch_length - len(input_ids)
            if pad_on_left:
                input_ids = ([pad_token] * padding_length) + input_ids
                attention_mask = ([0 if mask_padding_with_zero else 1] * padding_length) + attention_mask
            else:
                input_ids = input_ids + ([pad_token] * padding_length)
                attention_mask = attention_mask + ([0 if mask_padding_with_zero else 1] * padding_length)

            if len(input_ids) != batch_length:
                raise ValueError(f"Error with input length {len(input_ids)} vs {batch_length}")
            if len(attention_mask) != batch_length:
                raise ValueError(f"Error with input length {len(attention_mask)} vs {batch_length}")

            if self.mode == "classification":
                label = label_map[example.label]
            elif self.mode == "regression":
                label = float(example.label)
            else:
                raise ValueError(self.mode)

            if ex_index < 5 and self.verbose:
                logger.info("*** Example ***")
                logger.info(f"guid: {example.guid}")
                logger.info(f"input_ids: {' '.join([str(x) for x in input_ids])}")
                logger.info(f"attention_mask: {' '.join([str(x) for x in attention_mask])}")
                logger.info(f"label: {example.label} (id = {label})")

            features.append(InputFeatures(input_ids=input_ids, attention_mask=attention_mask, label=label))

        if return_tensors is None:
            return features
        elif return_tensors == "tf":
            if not is_tf_available():
                raise RuntimeError("return_tensors set to 'tf' but TensorFlow 2.0 can't be imported")
            import tensorflow as tf

            def gen():
                for ex in features:
                    yield ({"input_ids": ex.input_ids, "attention_mask": ex.attention_mask}, ex.label)

            dataset = tf.data.Dataset.from_generator(
                gen,
                ({"input_ids": tf.int32, "attention_mask": tf.int32}, tf.int64),
                ({"input_ids": tf.TensorShape([None]), "attention_mask": tf.TensorShape([None])}, tf.TensorShape([])),
            )
            return dataset
        elif return_tensors == "pt":
            if not is_torch_available():
                raise RuntimeError("return_tensors set to 'pt' but PyTorch can't be imported")
            import torch
            from torch.utils.data import TensorDataset

            all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
            all_attention_mask = torch.tensor([f.attention_mask for f in features], dtype=torch.long)
            if self.mode == "classification":
                all_labels = torch.tensor([f.label for f in features], dtype=torch.long)
            elif self.mode == "regression":
                all_labels = torch.tensor([f.label for f in features], dtype=torch.float)

            dataset = TensorDataset(all_input_ids, all_attention_mask, all_labels)
            return dataset
        else:


SUMMARY:
Here's a concise summary:

The `SingleSentenceClassificationProcessor` class is designed to process single sentence classification data. It provides methods for loading examples from CSV files, creating processors from text and labels, retrieving features with custom tokenizers, and converting the data into formats suitable for machine learning models.

Key features:

* Loads examples from CSV files with customizable column mappings
* Creates processors from text and labels arrays
* Retrieves task-specific input features with custom tokenizers
* Supports classification and regression tasks

The class uses the Hugging Face library to handle tokenization and feature generation. It provides flexibility in handling different data formats and model requirements.

This summary should provide a good overview of the class's functionality without requiring an extensive read of the entire codebase.


key: chroma:document
string_value:
CODE:

def glue_convert_examples_to_features(
    examples: Union[List[InputExample], "tf.data.Dataset"],
    tokenizer: PreTrainedTokenizer,
    max_length: Optional[int] = None,
    task=None,
    label_list=None,
    output_mode=None,
):
    """
    Loads a data file into a list of `InputFeatures`

    Args:
        examples: List of `InputExamples` or `tf.data.Dataset` containing the examples.
        tokenizer: Instance of a tokenizer that will tokenize the examples
        max_length: Maximum example length. Defaults to the tokenizer's max_len
        task: GLUE task
        label_list: List of labels. Can be obtained from the processor using the `processor.get_labels()` method
        output_mode: String indicating the output mode. Either `regression` or `classification`

    Returns:
        If the `examples` input is a `tf.data.Dataset`, will return a `tf.data.Dataset` containing the task-specific
        features. If the input is a list of `InputExamples`, will return a list of task-specific `InputFeatures` which
        can be fed to the model.

    """
    warnings.warn(DEPRECATION_WARNING.format("function"), FutureWarning)
    if is_tf_available() and isinstance(examples, tf.data.Dataset):
        if task is None:
            raise ValueError("When calling glue_convert_examples_to_features from TF, the task parameter is required.")
        return _tf_glue_convert_examples_to_features(examples, tokenizer, max_length=max_length, task=task)
    return _glue_convert_examples_to_features(
        examples, tokenizer, max_length=max_length, task=task, label_list=label_list, output_mode=output_mode


SUMMARY:
**Function Summary**

`glue_convert_examples_to_features`

Converts input data to model features for a GLUE task.

**Key Implementation Details**

* Takes in `examples`: either a list of `InputExamples` or a `tf.data.Dataset`
* Returns task-specific features as a `tf.data.Dataset` or a list of `InputFeatures`.

**Notable Dependencies and Requirements**

* `tokenizer`: instance of `PreTrainedTokenizer`
* `task`: GLUE task
* `label_list`: list of labels (optional)
* `output_mode`: string indicating output mode ("regression" or "classification") (optional)

**Method Signatures:**

* `_tf_glue_convert_examples_to_features(excperience, tokenizer, max_length)`
* `_glue_convert_examples_to_features(experiences, tokenizer, max_length, task=None, label_list=None, output_mode=None)`

**External Dependencies:**

* `tf.data.Dataset`: if `examples` is a dataset
* `input_example`, `pretrained_tokenizer`, `processor`: implied by the function call


key: chroma:document
string_value:
CODE:

def _glue_convert_examples_to_features(
    examples: List[InputExample],
    tokenizer: PreTrainedTokenizer,
    max_length: Optional[int] = None,
    task=None,
    label_list=None,
    output_mode=None,
):
    if max_length is None:
        max_length = tokenizer.model_max_length

    if task is not None:
        processor = glue_processors[task]()
        if label_list is None:
            label_list = processor.get_labels()
            logger.info(f"Using label list {label_list} for task {task}")
        if output_mode is None:
            output_mode = glue_output_modes[task]
            logger.info(f"Using output mode {output_mode} for task {task}")

    label_map = {label: i for i, label in enumerate(label_list)}

    def label_from_example(example: InputExample) -> Union[int, float, None]:
        if example.label is None:
            return None
        if output_mode == "classification":
            return label_map[example.label]
        elif output_mode == "regression":
            return float(example.label)
        raise KeyError(output_mode)

    labels = [label_from_example(example) for example in examples]

    batch_encoding = tokenizer(
        [(example.text_a, example.text_b) for example in examples],
        max_length=max_length,
        padding="max_length",
        truncation=True,
    )

    features = []
    for i in range(len(examples)):
        inputs = {k: batch_encoding[k][i] for k in batch_encoding}

        feature = InputFeatures(**inputs, label=labels[i])
        features.append(feature)

    for i, example in enumerate(examples[:5]):
        logger.info("*** Example ***")
        logger.info(f"guid: {example.guid}")
        logger.info(f"features: {features[i]}")



SUMMARY:
**`_glue_convert_examples_to_features` Function Summary**

*   **Purpose:** Converts examples to features for text classification or regression tasks using the given `PreTrainedTokenizer`.
*   **Functionality:**
    *   Tokenizes input strings and encodes into batch format
    *   Applies transformations (padding, truncation) according to max length
    *   Creates feature dictionary with labels from example attributes
    *   Returns a list of `InputFeatures` objects
*   **Key Features:**
    *   Utilizes `PreTrainedTokenizer` for tokenization and encoding
    *   Employs dynamic label mapping based on task-specific output modes
    *   Supports both classification and regression output modes

**Implementation Details:**

*   Uses `glue_processors` dictionary to retrieve specific processor object for the given task
*   Applies logging statements to display example GUIDs and features during development
*   Returns a list of `InputFeatures` objects in XML format for downstream model usage

**Notable Dependencies:**

*   Required: `PreTrainedTokenizer`
*   Optional:
    *   `glue_processors`
    *   `label_list`
    *   `output_mode`

**Interfaces and Inheritance:**
- Implements the `processor` interface from `glue_processors`, passing the task name as argument
- Extends the feature dictionary creation using `InputFeatures`, inheriting default implementation properties


key: chroma:document
string_value:
CODE:

class OutputMode(Enum):
    classification = "classification"


SUMMARY:
**Summary:**

This code defines an `Enum` called `OutputMode`, with two possible values: `classification` and `regression`. The primary purpose of this class is to specify the output mode for machine learning models.

**Key Implementation Details:**

*   The `Enum` class allows for a clear and concise representation of output modes.
*   The use of string literals ("classification" and "regression") as enum members ensures consistency in representing different output modes.

**Notable Dependencies or Requirements:**

*   Python's built-in `enum` module is not explicitly imported, but its functionality is utilized to define the `OutputMode` Enum.

Note that this summary focuses on the essential aspects of the code, excluding any external dependencies or additional implementation details that are not crucial to understanding the class's main purpose and functionality.


key: chroma:document
string_value:
CODE:

class MrpcProcessor(DataProcessor):
    """Processor for the MRPC data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["sentence1"].numpy().decode("utf-8"),
            tensor_dict["sentence2"].numpy().decode("utf-8"),
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        logger.info(f"LOOKING AT {os.path.join(data_dir, 'train.tsv')}")
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"{set_type}-{i}"
            text_a = line[3]
            text_b = line[4]
            label = None if set_type == "test" else line[0]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))


SUMMARY:
**Summary**

This `MrpcProcessor` class is designed to process the MRPC dataset for the GLUE benchmark. It inherits from `DataProcessor` and implements methods for reading example data from various file formats, creating input examples, and retrieving labels.

### Purpose and Functionality

* Processes MRPC dataset (GLUE version)
* Reads label encoding and conversion
* Handles file format reads (tsv)

### Key Implementation Details

* `_create_examples`
	+ Takes in a list of lines read from the tsv file and returns input examples for `train`, `dev`, or `test` sets.
* `_get_example_from_tensor_dict`
	+ Returns an `InputExample` object containing tensors, strings, and labels.

### Notable Dependencies and Requirements

* `DataProcessor`
* `os`
* `pathlib`
* `numpy`
* `pandas` (for `read_tsv`)
 

This class leverages the base functionality of `DataProcessor`, modifying it to handle the specific requirements of the MRPC dataset. The `_create_examples` method is a key part of the implementation, as it handles reading and converting the example data from the tsv files.


key: chroma:document
string_value:
CODE:

class MnliProcessor(DataProcessor):
    """Processor for the MultiNLI data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["premise"].numpy().decode("utf-8"),
            tensor_dict["hypothesis"].numpy().decode("utf-8"),
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev_matched.tsv")), "dev_matched")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test_matched.tsv")), "test_matched")

    def get_labels(self):
        """See base class."""
        return ["contradiction", "entailment", "neutral"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"{set_type}-{line[0]}"
            text_a = line[8]
            text_b = line[9]
            label = None if set_type.startswith("test") else line[-1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))


SUMMARY:
**MnliProcessor Class Summary**

The `MnliProcessor` class is a custom DataProcessor for the MultiNLI (Multi-Genre NLI) dataset, part of the GLUE benchmark. Its primary purpose is to load and preprocess data from the `train.tsv`, `dev_matched.tsv`, and `test_matched.tsv` files.

**Key Features:**

* Reads labels as either "contradiction", "entailment", or "neutral".
* Supports loading training, development, and testing sets.
* Preprocesses examples by converting tensor values to numpy arrays, decoding strings, and creating an InputExample instance for each line.

**Notable Dependencies:**

* `os` module for file path manipulation.
* `numpy` for data type manipulation.
* `FutureWarning` for deprecation warnings.

Note: The `DEPRECATION_WARNING` variable is not defined in this code snippet. It should be a predefined warning message to be used as a placeholder.


key: chroma:document
string_value:
CODE:

class MnliMismatchedProcessor(MnliProcessor):
    """Processor for the MultiNLI Mismatched data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev_mismatched.tsv")), "dev_mismatched")

    def get_test_examples(self, data_dir):
        """See base class."""


SUMMARY:
**Summary: MnliMismatchedProcessor Class**

The `MnliMismatchedProcessor` class is a custom processor for the MultiNLI Mismatched dataset in GLUE. It inherits from `MnliProcessor`.

**Primary Purpose:** To process and prepare data for training on the Multinational Lingo (MultiNLI) Mismatched dataset.

**Key Implementation Details:**

- `get_dev_examples`: Calls `_create_examples` with dev examples read from a TSV file (`dev_mismatched.tsv`) and labeled as "dev_mismatched".
- `get_test_examples`: Similar to above, but for test examples (`test_mismatched.tsv`).

**Dependencies/Requirements:**

- Uses the base class `MnliProcessor`.
- Reads data from files (`dev_mismatched.tsv` and `test_mismatched.tsv`) using `os.path.join`.
- Inherited from `MnliProcessor`, which likely includes additional requirements and dependencies.


key: chroma:document
string_value:
CODE:

class ColaProcessor(DataProcessor):
    """Processor for the CoLA data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["sentence"].numpy().decode("utf-8"),
            None,
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        test_mode = set_type == "test"
        if test_mode:
            lines = lines[1:]
        text_index = 1 if test_mode else 3
        examples = []
        for i, line in enumerate(lines):
            guid = f"{set_type}-{i}"
            text_a = line[text_index]
            label = None if test_mode else line[1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=None, label=label))


SUMMARY:
**ColaProcessor Class Summary**

The `ColaProcessor` class is a data processor for the CoLA (Communicating with Computers at Low Levels and Across Domains) dataset in the GLUE (General Language Understanding Evaluation) framework. It handles data loading and example creation for training, development, and testing sets.

**Main Functionality**

* Loads ColA dataset from TSV files
* Creates `InputExample` objects for each line of text
* Prepares examples for training, development, and testing sets

**Key Implementation Details**

* Uses `_create_examples` method to create example tuples with text A, label, and GUID
* Removes unnecessary rows from test set (excluding the first row)
* Appends GUID, text A, None (text B), and label to each example tuple

**Notable Dependencies and Requirements**

* `DataProcessor` class (base class for data processing tasks)
* `_read_tsv` function to read TSV files
* `InputExample` class to represent examples of text and labels


key: chroma:document
string_value:
CODE:

class Sst2Processor(DataProcessor):
    """Processor for the SST-2 data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["sentence"].numpy().decode("utf-8"),
            None,
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        examples = []
        text_index = 1 if set_type == "test" else 0
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"{set_type}-{i}"
            text_a = line[text_index]
            label = None if set_type == "test" else line[1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=None, label=label))


SUMMARY:
**Class Summary: Sst2Processor**

* **Purpose and Functionality**: A data processor for the SST-2 dataset in the GLUE benchmark. It reads example data from TSV files and prepares them for processing.

* **Key Implementation Details**:
	+ `_create_examples` method creates examples with 'guid', 'text_a'/'b', and 'label' attributes.
	+ Uses `InputExample` class to construct examples.
	+ Reads data from TSV files (`train.tsv`, `dev.tsv`, `test.tsv`) in the specified directory.

* **Notable Dependencies and Requirements**:
	+ Inherits from `DataProcessor`.
	+ Uses `os.path.join` and `warnings.warn`.
	+ References external libraries (`numpy`, `tsv`).
	+ Implements custom `InputExample` class.
	
* **Additional Notes**: This processor assumes that the TSV files are in specific format, with 'idx', 'sentence', 'label' columns.


key: chroma:document
string_value:
CODE:

class StsbProcessor(DataProcessor):
    """Processor for the STS-B data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["sentence1"].numpy().decode("utf-8"),
            tensor_dict["sentence2"].numpy().decode("utf-8"),
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")

    def get_labels(self):
        """See base class."""
        return [None]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"{set_type}-{line[0]}"
            text_a = line[7]
            text_b = line[8]
            label = None if set_type == "test" else line[-1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))


SUMMARY:
**STSBProcessor Class Summary**

### Main Purpose and Functionality
The `StsbProcessor` class is a data processor for the STS-B dataset (GLUE version), designed to read and preprocess dataset files.

### Key Implementation Details
- Creates `DataProcessor` instances using inherited methods.
- Uses `InputExample` objects to represent dataset examples.

### Notable Dependencies or Requirements
- `_read_tsv`: a method to read tab-separated values from files.
- `DEPRECATION_WARNING`: a string warning indicator for deprecated usage.

### MethodSignatures and Interfaces

* `get_example_from_tensor_dict(tensor_dict, str)` : takes in tensor dictionary object.
* `get_train_examples(data_dir, str)`: reads dataset from directory.
* `get_dev_examples(data_dir, str)`, `get_test_examples(data_dir, str)`: similar to get_train_examples but for dev and test sets.
* `_create_examples(lines, set_type, str)` : a private method for creating examples.


key: chroma:document
string_value:
CODE:

class QqpProcessor(DataProcessor):
    """Processor for the QQP data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["question1"].numpy().decode("utf-8"),
            tensor_dict["question2"].numpy().decode("utf-8"),
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        test_mode = set_type == "test"
        q1_index = 1 if test_mode else 3
        q2_index = 2 if test_mode else 4
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"{set_type}-{line[0]}"
            try:
                text_a = line[q1_index]
                text_b = line[q2_index]
                label = None if test_mode else line[5]
            except IndexError:
                continue
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))


SUMMARY:
**Class Summary: QqpProcessor**

The `QqpProcessor` class is a custom data processor for the QQP (Question Answering) dataset in GLUE. It inherits from `DataProcessor` and extends its functionality.

**Primary Purpose:** To process QQN data, converting it into an Input Dataset format.

**Main Functionality:**

* Constructs examples from training, dev, and test sets
* Retrieves and decodes text features from lines in the TSV files

**Key Implementation Details:**

* `_create_examples` function creates examples from training, dev, and test datasets
* Uses `InputExample` to create example objects with text-a, text-b, label, and guid attributes

**Notable Dependencies/Requirements:**

* Inherited from `DataProcessor`: `_read_tsv`, `os`
* Uses built-in Python types:
	+ numpy
	+ str
	+ list


key: chroma:document
string_value:
CODE:

class QnliProcessor(DataProcessor):
    """Processor for the QNLI data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["question"].numpy().decode("utf-8"),
            tensor_dict["sentence"].numpy().decode("utf-8"),
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")

    def get_labels(self):
        """See base class."""
        return ["entailment", "not_entailment"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"{set_type}-{line[0]}"
            text_a = line[1]
            text_b = line[2]
            label = None if set_type == "test" else line[-1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))


SUMMARY:
**Summary**

The QnliProcessor class is a custom data processor for the GLUE version of the QNLI (Question Natural Language Inference) dataset. It inherits from DataProcessor and provides functionality to load and process the dataset.

**Key Implementation Details**

* Processes examples from three sets: train, dev, and test
* Uses `_read_tsv` to read example lines from TSV files in the `data_dir`
* Creates `InputExample` objects with guided text, question, label, and more accurate processing based on set type
* Provides labels for entailment and not entailment classes

**Notable Dependencies/Requirements**

* Inherited interfaces: DataProcessor
* External dependencies:
	+ os.path.join to access file paths
	+ str.decode("utf-8") to convert tensor data to strings


key: chroma:document
string_value:
CODE:

class RteProcessor(DataProcessor):
    """Processor for the RTE data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["sentence1"].numpy().decode("utf-8"),
            tensor_dict["sentence2"].numpy().decode("utf-8"),
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")

    def get_labels(self):
        """See base class."""
        return ["entailment", "not_entailment"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"{set_type}-{line[0]}"
            text_a = line[1]
            text_b = line[2]
            label = None if set_type == "test" else line[-1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))


SUMMARY:
**Class Summary: RteProcessor**

**Purpose and Functionality**
---------------------------

The `RteProcessor` class is a data processor for the RTE (Reasoning over Text in New Datasets) dataset on the GLUE benchmark. It inherits from the `DataProcessor` class and provides methods to preprocess and create examples for training, development, and testing sets.

**Key Implementation Details**
-----------------------------

* The `get_example_from_tensor_dict` method converts a tensor dictionary into an `InputExample` object.
* The `get_train_examples`, `get_dev_examples`, and `get_test_examples` methods read TSV files from the given data directory and create examples using the `_create_excelamples` method.
* The `_create_examples` method creates individual examples from each line in the TSV file, including text A, B, and label.

**Notable Dependencies or Requirements**
---------------------------------------

* `DataProcessor` class (base class)
* `InputExample` class
* `os.path.join`
* `string.decode`

Note that this summary focuses on the key aspects of the class, excluding unnecessary details. The provided code chunk has been analyzed to extract relevant information and provide a concise summary.


key: chroma:document
string_value:
CODE:

class WnliProcessor(DataProcessor):
    """Processor for the WNLI data set (GLUE version)."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        warnings.warn(DEPRECATION_WARNING.format("processor"), FutureWarning)

    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["sentence1"].numpy().decode("utf-8"),
            tensor_dict["sentence2"].numpy().decode("utf-8"),
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training, dev and test sets."""
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = f"{set_type}-{line[0]}"
            text_a = line[1]
            text_b = line[2]
            label = None if set_type == "test" else line[-1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))


SUMMARY:
**Summary: WnliProcessor Class**

The `WnliProcessor` class inherits from `DataProcessor` and processes the WNLI data set (GLUE version). It provides methods for:

* Loading training, dev, and test examples from dataset files (`tsv`)
* Creating `InputExample` instances with sentence pairs and labels

**Key Implementation Details:**

* `_create_examples` method creates training, dev, and test examples by iterating through lines of datasets
* Uses pre-defined label list `["0", "1"]` for training data
* Utilizes `_read_tsv` method to read tsv files from dataset directories

**Notable Dependencies and Requirements:**

* `_read_tsv` function to read tsv files (assumed to be defined elsewhere)
* `DataProcessor` base class (implemented elsewhere)
* GLUE version of WNLI data set
* `InputExample` class (impluded elsewhere)

**Key Method Signatures:**

* `get_example_from_tensor_dict`
* `_create_examples`
* `_read_tsv`

This summary provides an overview of the main functionality, key implementation details, and notable dependencies required to understand the WnliProcessor class.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/src/transformers/utils/dummy_timm_and_torchvision_objects.py


key: chroma:document
string_value:
CODE:

class TimmWrapperImageProcessor(metaclass=DummyObject):
    _backends = ["timm", "torchvision"]

    def __init__(self, *args, **kwargs):


SUMMARY:
**Summary:**

`TimmWrapperImageProcessor` is an image processing class that uses two backend libraries (`timm` and `torchvision`) to enhance its functionality. The constructor initializes the object with required backends using `requires_backends` method, ensuring at least one of them is present.

*   **Main Purpose and Functionality:** Image processing utility.
*   **Important Implementation Details:**
    *   Constructor (`__init__`) ensures presence of either `timm` or `torchvision` backend.
*   **Notable Dependencies or Requirements:**
    *   `timm` backend
    *   `torchvision`
        Requires both at least one of these backends to be present.


key: docstring
string_value:
# pylint: disable=invalid-name
# TODO: This doesn't work for all packages (`bs4`, `faiss`, etc.) Talk to Sylvain to see how to do with it better.


key: chroma:document
string_value:
CODE:
# TODO: This doesn't work for all packages (`bs4`, `faiss`, etc.) Talk to Sylvain to see how to do with it better.
def _is_package_available(pkg_name: str, return_version: bool = False) -> Union[Tuple[bool, str], bool]:
    # Check if the package spec exists and grab its version to avoid importing a local directory
    package_exists = importlib.util.find_spec(pkg_name) is not None
    package_version = "N/A"
    if package_exists:
        try:
            # Primary method to get the package version
            package_version = importlib.metadata.version(pkg_name)
        except importlib.metadata.PackageNotFoundError:
            # Fallback method: Only for "torch" and versions containing "dev"
            if pkg_name == "torch":
                try:
                    package = importlib.import_module(pkg_name)
                    temp_version = getattr(package, "__version__", "N/A")
                    # Check if the version contains "dev"
                    if "dev" in temp_version:
                        package_version = temp_version
                        package_exists = True
                    else:
                        package_exists = False
                except ImportError:
                    # If the package can't be imported, it's not available
                    package_exists = False
            else:
                # For packages other than "torch", don't attempt the fallback and set as not available
                package_exists = False
        logger.debug(f"Detected {pkg_name} version: {package_version}")
    if return_version:
        return package_exists, package_version
    else:


SUMMARY:
**Summary**

### _is_package_available Function

#### Purpose

Determines whether a package is available and, if so, returns its version.

#### Functionality

`_is_package_available` checks the existence of a package by attempting to import it using `importlib.import_module`. If successful, it retrieves the package's version using `importlib.metadata.version`.

**Key Implementation Details**

*   Checks for `torch` package with fallback logic
*   Uses `importlib.util.find_spec` to check if package spec exists
*   Returns a tuple containing availability and package version (if requested)

#### Notable Dependencies and Requirements

*   `importlib` module
*   `logger` object (used for debugging logs)

**Key Method Signatures and Interfaces**

*   `_is_package_available(pkg_name: str, return_version: bool = False) -> Union[Tuple[bool, str], bool]`

#### External Dependencies

*   `importlib`
*   `importlib.metadata`

The function is well-structured, but it might benefit from additional checks or handling for certain packages as suggested in the TODO comment.


key: chroma:document
string_value:
CODE:

def is_kenlm_available():


SUMMARY:
Here is a concise summary:

**Function Summary:** `is_kenlm_available()`

* **Purpose/Functionality:** Checks the availability of `_kenlm_available`.
* **Key Implementation Details:**
	+ Returns the value of `_kenlm_available` directly, indicating no additional logic or decision-making involved.
* **Notable Dependencies:**
	+ None explicitly listed, but references a private variable `_kenlm_available`.

**Note:** The function seems to be a simple wrapper or alias, potentially serving as a guard clause to determine the availability of a specific external library (`kenlm`) in the codebase.


key: chroma:document
string_value:
CODE:

def is_cv2_available():


SUMMARY:
**Summary:**

`is_cv2_available` function:

* Returns the status of OpenCV 2.4.8 availability (`_cv2_available`).
* No specific implementation details or key algorithms are present in this snippet.

**Key Pointers:**

* `_cv2_available`: Reference to a global variable indicating OpenCV 2.4.8 availability.
* External dependency: `_cv2_available` is assumed to be defined elsewhere, likely in a parent class or module (`opencv.py`).


key: chroma:document
string_value:
CODE:

def is_torch_available():


SUMMARY:
**Summary:**

`is_torch_available()` Function Summarization:

* **Main Purpose:** Determines whether the Torch library is available for use.

**Key Details:**

* Returns the result of `_torch_available`, indicating availability status.

**Notable Dependency:**

* `_torch_available`: an existing variable (not shown in this code snippet) that likely evaluates Torch's availability.
 
**Interface/Method Signature:** `is_torch_available()`
```markdown
# is_torch_available()
return _torch_available
```


key: chroma:document
string_value:
CODE:

def is_accelerate_available(min_version: str = ACCELERATE_MIN_VERSION):


SUMMARY:
**Summary**

This function (`is_accelerate_available`) checks if the system is compatible with an accelerate module.

**Key Points:**

* Verifies if `_accelerate_available` is set to `True`.
* Compares the target version (`_accelerate_version`) with a minimum required version (`min_version`) using the `version.parse()` method.
* Returns `True` if the target version meets or exceeds the minimum required version.

**Implementation Details:**

* The function uses `_ACCELERATE_MIN_VERSION` as a default value for the `min_version` parameter.
* The comparison is performed between two `str` versions parsed using the `version.parse()` method.


key: chroma:document
string_value:
CODE:

def is_torch_deterministic():
    """
    Check whether pytorch uses deterministic algorithms by looking if torch.set_deterministic_debug_mode() is set to 1 or 2"
    """
    import torch

    if torch.get_deterministic_debug_mode() == 0:
        return False
    else:


SUMMARY:
**Function Summary: `is_torch_deterministic()`**

The `is_torch_deterministic()` function checks whether PyTorch uses deterministic algorithms by checking the value of `torch.set_deterministic_debug_mode()`. If set to 0, it returns False; otherwise, it returns True.

**Key Details:**

* Function name: `is_torch_deterministic()`
* Purpose: Check PyTorch's determinism
* Returns:
	+ False if `torch.set_deterministic_debug_mode()` is 0
	+ True otherwise

**Notable Dependencies:**
- `torch`: The PyTorch library


key: chroma:document
string_value:
CODE:

def is_hadamard_available():


SUMMARY:
**Summary:**

### Function Analysis

The `is_hadamard_available()` function checks the availability of Hadamard matrix functionality.

#### KeyDetails:

* Returns a boolean value indicating whether Hadamard available is true or false (`_hadamard_available`).
* Method name closely matches the purpose (`is_*` convention often used for query-like methods).

#### NotableDependency:

* `_hadamard_available`: an internal variable representing the availability of Hadamard matrix functionality.


key: chroma:document
string_value:
CODE:

def is_hqq_available(min_version: str = HQQ_MIN_VERSION):


SUMMARY:
**Summary:**

This function, `is_hqq_available`, checks if the `Hqq` feature is available in the system by comparing the system's `_hqq_available` flag with the version of `(_hqq_version)` against the minimum required version specified by `_min_version`. If both conditions are met, it returns `True`; otherwise, it returns `False`.

**Key Details:**

* Checks if Hqq feature is enabled (`_hqq_available`)
* Compares system's _hqq_version against minimum required version (`_min_version`) using a version parser (`version.parse()`)
* Returns `True` if both conditions are met; otherwise, returns `False`

(Note: `HQQ_MIN_VERSION`, `_hqq_available`, `_hqq_version`, and `version` are assumed to be defined elsewhere in the codebase.)


key: chroma:document
string_value:
CODE:

def is_pygments_available():


SUMMARY:
**Summary**

This function checks the availability of Pygments, a third-party dependency. It simply returns a cached result of `_pygments_available`, indicating that the implementation details are not directly provided in this snippet.

**Key Points:**

* Primary purpose: Check Pygments availability
* Implementation details:
	+ Returns cached result of `_pygments_available`
* Notable dependencies/requirements:
	+ `Pygments` library (not explicitly imported, assumed to be a dependency of the project)
* Inherited/implemented interfaces: None


key: chroma:document
string_value:
CODE:

def get_torch_version():


SUMMARY:
**Torch Version Retrieval Function**

* **Purpose**: Retrieves the version of PyTorch stored in `_torch_version`.
* **Implementation Details**:
	+ Returns the value of `_torch_version` directly using Duck Typing (`return _torch_version`)
* **Notable Dependency/Requirement**: Uses a private variable (`_torch_version`) which is likely set elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def is_torch_sdpa_available():
    if not is_torch_available():
        return False
    elif _torch_version == "N/A":
        return False

    # NOTE: We require torch>=2.1 (and not torch>=2.0) to use SDPA in Transformers for two reasons:
    # - Allow the global use of the `scale` argument introduced in https://github.com/pytorch/pytorch/pull/95259
    # - Memory-efficient attention supports arbitrary attention_mask: https://github.com/pytorch/pytorch/pull/104310
    # NOTE: MLU is OK with non-contiguous inputs.
    if is_torch_mlu_available():
        return version.parse(_torch_version) >= version.parse("2.1.0")
    # NOTE: NPU can use SDPA in Transformers with torch>=2.1.0.
    if is_torch_npu_available():
        return version.parse(_torch_version) >= version.parse("2.1.0")
    # NOTE: We require torch>=2.1.1 to avoid a numerical issue in SDPA with non-contiguous inputs: https://github.com/pytorch/pytorch/issues/112577


SUMMARY:
**Summary**

This function checks if Torch SDPA is available for use in Transformers.

### Purpose and Functionality

The `is_torch_sdpa_available()` function determines the availability of Torch SDPA (Scaling Down Pad Algorithm) based on the version of PyTorch installed. It returns `False` if PyTorch is not compatible or its version does not meet specific requirements.

**Key Implementation Details**

* The function uses conditional checks to ensure compatibility with various PyTorch versions.
* It requires at least PyTorch 2.1 for SDPA functionality, but recommends version 2.1.0 and 2.1.1 for optimal performance.
* The `is_torch_mlu_available()` and `is_torch_npu_available()` functions are called to determine availability in MLU (Tensor Cores) and NPU architectures, respectively.

### Notable Dependencies or Requirements

* PyTorch version 2.0 or greater is not supported for SDPA usage.
* Specific versions of PyTorch (2.1.0 and later) are required for optimal performance on non-contiguous inputs.
* External dependencies:
	+ `torch` module
	+ `version` module


key: chroma:document
string_value:
CODE:

def is_torch_flex_attn_available():
    if not is_torch_available():
        return False
    elif _torch_version == "N/A":
        return False

    # TODO check if some bugs cause push backs on the exact version
    # NOTE: We require torch>=2.5.0 as it is the first release


SUMMARY:
**Summary**

This function checks if the TorchFlex attention module is available and compatible with at least version 2.5.0 of PyTorch.

**Key Points**

* Verifies availability using `is_torch_available()`
* Checks for valid version strings (`_torch_version`) using the `version.parse()` method
* Requires PyTorch version >= 2.5.0

**Implementation Details**

* Calls `_torch_version` with a `TODO` comment to indicate potential bug impact on exact version checks
* Utilizes `version.parse()` to parse and compare version strings for compatibility check


key: chroma:document
string_value:
CODE:

def is_torchvision_available():


SUMMARY:
**Summary:**

The `is_torchvision_available()` function is a wrapper that simply returns the value of `_torchvision_available`, indicating whether the PyTorch Vision module is available.

**Key Details:**

* Returns a boolean value from `_torchvision_available`
* No additional implementation or logic present
* Purpose: Determine availability of PyTorch Vision

**Notable Dependencies/Requirements:**

* None explicitly stated, but it's implied that the `_torchvision_available` variable is defined elsewhere in the codebase.
* Notably references internal variables without external dependencies.


key: chroma:document
string_value:
CODE:

def is_torchvision_v2_available():
    if not is_torchvision_available():
        return False

    # NOTE: We require torchvision>=0.15 as v2 transforms are available from this version: https://pytorch.org/vision/stable/transforms.html#v1-or-v2-which-one-should-i-use


SUMMARY:
**Summary:**

The `is_torchvision_v2_available` function checks if torchvision version 2 is available by verifying the existence of torchvision in general and then checking if the version is greater than or equal to 0.15.

**Key Points:**

* **Purpose:** Checks availability of torchvision v2.
* **Implementation Details:**
	+ Uses `is_torchvision_available` utility function.
	+ Requires torchvision>=0.15 for v2 transforms.
* **Dependencies:**
	+ `_torchvision_version`: expected to be a version string parsed using the `version.parse` method.

**Interface:** Returns a boolean indicating whether torchvision v2 is available.


key: chroma:document
string_value:
CODE:

def is_galore_torch_available():


SUMMARY:
**Summary:**

The `is_galore_torch_available` function returns a boolean value indicating whether the `_galore_torch_available` flag is set, likely controlling access to the `galore_torch` library.

**Key details:**

* Returns a boolean value (`True` or `False`)
* Exposes the `_galore_torch_available` internal flag
* Purpose: Control access to `galore_torch` library

No external dependencies or notable interfaces.


key: chroma:document
string_value:
CODE:

def is_lomo_available():


SUMMARY:
**Summary:**

```markdown
# `is_lomo_available` Function Summary
=============================================

### Purpose and Functionality

Returns the availability of LOMO (likely a third-party service or library).

### Key Implementation Details

* Returns a boolean value indicating availability.

### Notable Dependencies/Requirements

* `_lomo_available`: a pre-defined, internal constant.
```

Note: The function name `is_lomo_available` and the return statement are not showing the actual implementation of `_lomo_available`, but assuming that it is already defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def is_grokadamw_available():


SUMMARY:
**Summary**

The `is_grokadamw_available()` function checks the availability of the `_grokadamw_available` constant. It simply returns its value, indicating whether GrokadaMw is enabled or not.

**Key Details:**

* No key algorithms, patterns, or design choices are applied in this implementation.
* Returns a boolean value `True` if `_grokadamw_available` is truthy, and `False` otherwise.
* No external dependencies or interfaces are used.


key: chroma:document
string_value:
CODE:

def is_schedulefree_available():


SUMMARY:
**Summary:**

This Function `is_schedulefree_available()` 

* **Primary Purpose:** Returns a boolean value indicating availability of schedulefree.
* **Key Implementation Details:**
	+ Wraps direct access to `_schedulefree_available` return value.
* **Notable Dependencies:**
	+ Internal variable `_schedulefree_available`.
	+ No external dependencies referenced by exact name.


key: chroma:document
string_value:
CODE:

def is_pyctcdecode_available():


SUMMARY:
**Summary:**

The `_is_pyctcdecode_available` function checks if the `_pyctcdecode` variable is defined and returns its availability.

*   **Purpose:** To determine whether the `_pyctcdecode` feature is available or not.
*   **Key Implementation Details:** Directly returning the value of `_pyctcdecode_available`.
*   **Notable Dependencies:** None explicitly mentioned, but implies access to global variables (`_pyctcdecode`, `_pyctcdecode_available`).
*   **Required Interfaces:** None.

**RAG System Summary:**

`is_pyctcdecode_available()`: Returns availability of "_pyctcdecode" feature.

*   Availability: `'_pyctcdecode_available'`

Note: `_pyctcdecode_available` is not defined in the given function code chunk, suggesting it might be a global variable or something to be imported.


key: chroma:document
string_value:
CODE:

def is_librosa_available():


SUMMARY:
**Summary:**

The `is_librosa_available` function checks the availability of the Librosa library.

**Key Details:**

* Returns a boolean value indicating whether `_librosa_available` is truthy.
* No specific algorithm or pattern is used.

**Note:** The implementation simply returns another variable, `_librosa_available`, without any conditional checking.

**Context:** This function likely serves as a singleton check for the Librosa library's availability.


key: chroma:document
string_value:
CODE:

def is_essentia_available():


SUMMARY:
**Summary: `is_essentia_available()` Function**

This utility function checks for the availability of the Essentia library.

**Key Implementation Details:**

*   Returns a boolean value (`true`/`false`) indicating whether Essentia is available.
*   Utilizes an internal variable `_essentia_available` to determine its availability.

**Notable Dependencies or Requirements:**

*   This function relies on the existence of an internal variable `_essentia_available`, which is not explicitly defined in this snippet. It is assumed that this variable exists elsewhere in the codebase and holds the current status of Essentia's availability.

**RAG System Tags:**
*   Purpose: Utility Function
*   Visibility: Private (Local)
*   Dependency: `_essentia_available`


key: chroma:document
string_value:
CODE:

def is_pretty_midi_available():


SUMMARY:
**Summary**

`is_pretty_midi_available()` function checks if the `_pretty_midi_available` variable is truthy, implicitly indicating the availability of the PrettyMIDI library.

**Key Implementation Details:**

* Returns the value of `_pretty_midi_available`
* No custom logic or algorithm employed
* External dependency on `PrettyMIDI` library


key: chroma:document
string_value:
CODE:

def is_torch_cuda_available():
    if is_torch_available():
        import torch

        return torch.cuda.is_available()
    else:


SUMMARY:
**Function Summary: `is_torch_cuda_available`**

**Purpose:** Determines whether a Torch CUDA driver is available.
 
- **Primary Purpose:** Checks for availability of a compatible Python environment.

 **Implementation Details:**
 
* Uses `torch.cuda.is_available()` to verify CUDA support, leveraging the availability of a valid CPU torch instance on failure.

**Notable Dependencies/Requirements:**

*   `torch` (imported if Torch is available)


key: chroma:document
string_value:
CODE:

def is_mamba_ssm_available():
    if is_torch_available():
        import torch

        if not torch.cuda.is_available():
            return False
        else:
            return _is_package_available("mamba_ssm")


SUMMARY:
**Summary: `is_mamba_ssm_available` Function**

The `is_mamba_ssm_available` function checks if the Mamba SSM package is available on the system with CUDA support using PyTorch.

### Purpose and Functionality
- Primary purpose: Verifies availability of Mamba SSM package with CUDA support.
- Technical terms used: Mamba, SSM (Signal Separation Model), CUDA, PyTorch.

### Key Implementation Details

*   Checks for PyTorch availability and CUDA support using `is_torch_available()` and `torch.cuda.is_available()`.
*   Verifies Mamba SSM package availability by calling `_is_package_available("mamba_ssm")`.

### Notable Dependencies or Requirements
- **PyTorch**: Imported to utilize CUDA functionality.
- **mamba-ssm package**: Required for Mamba SSM functionality.
- No external dependencies listed.


key: chroma:document
string_value:
CODE:

def is_mamba_2_ssm_available():
    if is_torch_available():
        import torch

        if not torch.cuda.is_available():
            return False
        else:
            if _is_package_available("mamba_ssm"):
                import mamba_ssm

                if version.parse(mamba_ssm.__version__) >= version.parse("2.0.4"):
                    return True


SUMMARY:
**Summary**

The `is_mamba_2_ssm_available` function checks if the Mamba SSM (Software Sales Management) package is available and compatible with PyTorch version 2.0.4 or higher.

**Key Features**

* Checks availability of torch.cuda
* Verifies compatibility with mamba_ssm package version ≥ 2.0.4
* Imports required modules (torch, mamba_ssm) only when necessary

**Notable Dependencies/Interfaces**

* `torch`
* `_is_package_available("mamba_ssm")` interface
* `version.parse()` function

**Return Value**

`True` if Mamba SSM package is available and compatible; `False` otherwise.


key: chroma:document
string_value:
CODE:

def is_causal_conv1d_available():
    if is_torch_available():
        import torch

        if not torch.cuda.is_available():
            return False
        return _is_package_available("causal_conv1d")


SUMMARY:
**Function Summary**

`is_causal_conv1d_available()` checks if the `causal_conv1d` package is installed and available on the CUDA device.

**Key Details**

*   Checks for Torcha installation using `is_torch_available()`.
*   Verifies CUDA availability using `torch.cuda.is_available()`.
*   Calls `_is_package_available("causal_conv1d")` to check if the `causal_conv1d` package is installed.
*   Returns False if no Torcha installation or CUDA device is available.


key: chroma:document
string_value:
CODE:

def is_mambapy_available():
    if is_torch_available():
        return _is_package_available("mambapy")


SUMMARY:
**Summary:**

Ensures MambaPy availability by checking Torch's presence and attempting to verify package availability.

**Key Elements:**

* Utilizes `is_torch_available()` function to verify Torch's presence
* Calls `_is_package_available("mambapy")` if Torch is available, returning the result
* Returns `False` in all other cases

**Notable Dependencies/Requirements:**

* Torch (verified by `is_torch_available()`)
* `_is_package_available()` function (required for MambaPy verification)
         


key: chroma:document
string_value:
CODE:

def is_torch_mps_available(min_version: Optional[str] = None):
    if is_torch_available():
        import torch

        if hasattr(torch.backends, "mps"):
            backend_available = torch.backends.mps.is_available() and torch.backends.mps.is_built()
            if min_version is not None:
                flag = version.parse(_torch_version) >= version.parse(min_version)
                backend_available = backend_available and flag
            return backend_available


SUMMARY:
**Summary:**

`is_torch_mps_available` function checks whether Torch's MPS (Mixed Precision Training) backend is available.

**Key Concepts:**

* [Torch](https://pytorch.org/) availability check using `torch.backends.mps`.
* Minimum version requirement (`min_version`) for MPS to be enabled.
* Backend availability and built status (`is_available` and `is_built`) checked against the minimum version.

**Notable Implementation Details:**

* Import of Torch and its dependencies.
* Use of `_torch_version` for parsing comparison with `min_version`.

**Return Value:** `True` if MPS is available, `False` otherwise.


key: chroma:document
string_value:
CODE:

def is_torch_bf16_gpu_available():
    if not is_torch_available():
        return False

    import torch



SUMMARY:
**Summary:**

Checks availability of Torch 16-bit Binary Floating Point (BF16) GPU support.

*   **Purpose:** Verifies if BF16 support is available on the CUDA device.
*   **Key Implementation Details:**
    *   Uses `is_torch_available()` helper function to check for Torch presence.
    *   Imports `torch` module and queries cuda availability with `torch.cuda.is_available()`.
    *   Checks specific BF16 support using `torch.cuda.is_bf16_supported()`.
*   **Notable Dependencies and Requirements:**
    *   `torch` module
    *   CUDA device support


key: chroma:document
string_value:
CODE:

def is_torch_bf16_cpu_available():
    if not is_torch_available():
        return False

    import torch

    try:
        # multiple levels of AttributeError depending on the pytorch version so do them all in one check
        _ = torch.cpu.amp.autocast
    except AttributeError:
        return False



SUMMARY:
**Summary**

 Checks if PyTorch BF16 is available on CPU and returns `True` if `torch` module and `autocast` function are present.

**Key Points:**

*   Main Purpose: Verifies availability of PyTorch BF16 feature on CPU
*   Functionality:
    +   Imports the `torch` module
    +   Tries to use the `autocast` function from `torch.cpu.amp`
*   Narrowed Down Details:
    +   Used a try-except block with multiple possible `AttributeError` reasons related to PyTorch versions.
    +   Method Signature: Not applicable (function definition)
*   Notable Dependencies or Requirements:
    +   `torch`, `torch.cpu.amp`
    *External Dependency:* Not listed separately, but the availability of this function relies on external dependencies like `pytorch`.

**RAG System Representation**

```
Module: torch
Function: is_torch_bf16_cpu_available
Return Value: Availability flag

Description:
Verifies presence of PyTorch BF16 feature on CPU.
```


key: chroma:document
string_value:
CODE:

def is_torch_bf16_available():
    # the original bf16 check was for gpu only, but later a cpu/bf16 combo has emerged so this util
    # has become ambiguous and therefore deprecated
    warnings.warn(
        "The util is_torch_bf16_available is deprecated, please use is_torch_bf16_gpu_available "
        "or is_torch_bf16_cpu_available instead according to whether it's used with cpu or gpu",
        FutureWarning,
    )


SUMMARY:
**Summary**

The `is_torch_bf16_available()` function deprecates the original check for Torchaudio BF16 support, now only checking on GPU availability. It returns the result of `is_torch_bf16_gpuAvailable()`. 

**Primary Purpose**: To indicate deprecation and facilitate upgrade to more specific checks.

**Key Implementation Details**:

* Calls `warnings.warn()` to highlight deprecation.
* Returns the result of `is_torch_bf16_gpu_available()` (not shown in this snippet).

**Notable Dependencies or Requirements**: None, but it depends on `torchaudio` and its internal implementations.


key: chroma:document
string_value:
CODE:

def is_torch_tf32_available():
    if not is_torch_available():
        return False

    import torch

    if not torch.cuda.is_available() or torch.version.cuda is None:
        return False
    if torch.cuda.get_device_properties(torch.cuda.current_device()).major < 8:
        return False
    if int(torch.version.cuda.split(".")[0]) < 11:
        return False
    if version.parse(version.parse(torch.__version__).base_version) < version.parse("1.7"):
        return False



SUMMARY:
**Summary:**

The `is_torch_tf32_available` function checks if Torch TF32 (TensorFlow 32-bit integers) is available on the system.

**Key Implementation Details:**

*   Checks for availability of GPU support and compatible device properties
*   Verifies version numbers (`torch.cuda` and Torch versions)
*   Ensures compatibility with TF32 features using version comparisons

**Return Value:** `True` if TF32 is available, `False` otherwise.


key: chroma:document
string_value:
CODE:

def is_torch_fx_available():


SUMMARY:
**Summary**

`is_torch_fx_available()` checks the availability of TorchFX (Torch's high-performance execution model).

**Key Details**

* Returns the result of `_torch_fx_available`, indicating whether TorchFX is available (`True`) or not (`False`).


key: chroma:document
string_value:
CODE:

def is_peft_available():


SUMMARY:
**Summary:**

`is_peft_available()` Function

* **Purpose:** Returns a boolean indicating whether the `_peft_available` variable is set.

**Key Details:**

- Variable (`_peft_available`) accessed directly, implying a global or class-level scope.
- Direct return of primitive value without processing or transformation.


key: chroma:document
string_value:
CODE:

def is_bs4_available():


SUMMARY:
**Summary:**

* **Purpose:** Returns whether the `bs4` library (specifically, `_bs4_available`) is available or not.
* **Functionality:** Simple wrapper around the private variable `_bs4_available`, returning its value directly.
* **Key Implementation Details:** None.
* **Notable Dependency:** `_bs4_available` (internal state).

No external dependencies were found.


key: chroma:document
string_value:
CODE:

def is_tf_available():


SUMMARY:
**Summary**

The `is_tf_available` function returns a boolean value indicating whether TensorFlow is available.

**Key Details:**

* Retries calling the `_tf_available` function (not shown in this snippet)
* Returns: Boolean result (`return _tf_available`)

**Note:** The implementation of `_tf_available` is not provided, as it is assumed to be defined elsewhere.


key: chroma:document
string_value:
CODE:

def is_coloredlogs_available():


SUMMARY:
**Summary:**

`is_coloredlogs_available()` checks if the `_coloredlogs_available` attribute is defined.

**Key Details:**

* Returns whether `_coloredlogs_available` attribute is set to true or false (exact value not checked).
* Implies dependency on external variable or attribute initialization.
* Does not perform any actual logging-related operations.

**Notable Dependencies/Interfaces:** `_coloredlogs_available`


key: chroma:document
string_value:
CODE:

def is_tf2onnx_available():


SUMMARY:
**Summary:**

The `is_tf2onnx_available()` function returns a boolean indicating whether the TF2-ONNX availability check has been performed, effectively stubbing the implementation. 

**Key Details:**

* It wraps around `_tf2onnx_available`, which is assumed to be the actual availability check implementation.
* The primary purpose appears to be allowing for conditional checks or error handling with a minimal code footprint.


key: chroma:document
string_value:
CODE:

def is_onnx_available():


SUMMARY:
**Summary**

Convenience wrapper around the `is_onnx_available` check. Returns a boolean indicating whether ONNX (Open Neural Network Exchange) is available.

* **Purpose**: Provide an easy-to-use interface to determine if ONNX is supported.
* **Key implementation details**: None, simply delegates to `_onnx_available`.
* **Notable dependencies**: - `_onnx_available`: A private function checking for ONNX availability.


key: chroma:document
string_value:
CODE:

def is_openai_available():


SUMMARY:
**Summary:**

Function `is_openai_available`: Returns a boolean value indicating availability of OpenAI services.

**Details:**

* Primary purpose: Checks if OpenAI API is available for access.
* Key implementation detail:
	+ Returns `_openai_available`, suggesting a centralized flag controlling OpenAI API access.
* Notable dependencies or requirements:
	+ `_openai_available` (likely a global variable or constant).
* External dependency:
	+ None mentioned, but may depend on `OpenAI API` (unspecified).


key: chroma:document
string_value:
CODE:

def is_flax_available():


SUMMARY:
**Summary**

The `is_flax_available()` function checks and returns whether the Flax library (`_flax_available`) is available for use.

**Key Implementation Details**

* Returns a boolean value indicating availability ([True/False])
* Utilizes a static variable `_flax_available` as a flag [type of variable: `bool`]

**Notable Dependencies or Requirements**

* None apparent, but implies access to the Flax library.


key: chroma:document
string_value:
CODE:

def is_flute_available():
    try:
        return importlib.util.find_spec("flute") is not None and importlib.metadata.version("flute-kernel") >= "0.3.0"
    except importlib.metadata.PackageNotFoundError:


SUMMARY:
**Function Summary: `is_flute_available()`**

**Purpose:** Checks if the Flute library is available and its kernel version meets a minimum requirement.

**Details:**

* Attempts to find the "flute" package using `importlib.util.find_spec()`.
* If successful, checks the "flute-kernel" version using `importlib.metadata.version()` against "0.3.0".
* Returns `True` if Flute is available with kernel version >= 0.3.0, otherwise `False`.

**Notable Dependencies:**

* `importlib.util`
* `importlib.metadata`

**Exceptions:**

* `PackageNotFoundError`: raised if the "flute" package is not found.


key: chroma:document
string_value:
CODE:

def is_ftfy_available():


SUMMARY:
**Summary:**

This code snippet defines a function `is_ftfy_available()` that returns the value of the `_ftfy_available` constant, indicating whether FTFY (Fast and Friendly Translation Utility) is available.

**Analysis:**

* **Purpose:** Returns availability status of FTFY.
* **Implementation Details:**
	+ Uses a private constant `_ftfy_available`.
	+ Directly returns this constant without any additional logic.
* **Dependencies/Requirements:**
	+ None explicitly stated; assumes the constant is defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def is_g2p_en_available():


SUMMARY:
**Summary:**

`is_g2p_en_available`: A read-only utility function that returns whether the `_g2p_en_available` flag is set, indicating availability of the English variant of the Google Transliterate library.

**Key Implementation Details:**

* The implementation simply returns a cached value `_g2p_en_available`, implying that this check can be performed without actual initialization or computation.

**Notable Dependencies:**

* None explicitly mentioned.
However, the function relies on the `_g2p_en_available` flag which is assumed to be defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def is_torchdynamo_available():
    if not is_torch_available():
        return False



SUMMARY:
**Summary:**

Function `is_torchdynamo_available()` checks if TorchDynamo is available by verifying the availability of PyTorch and compliance with version 2.0.0 or later.

**Key Details:**

* Checks PyTorch availability using function `is_torch_available()`
* Verifies if `_torch_version` meets or exceeds version 2.0.0

**External Dependencies:**

* PyTorch (`version` is assumed to be from the `version` module in PyTorch)


key: chroma:document
string_value:
CODE:

def is_torch_compile_available():
    if not is_torch_available():
        return False

    import torch

    # We don't do any version check here to support nighlies marked as 1.14. Ultimately needs to check version against
    # 2.0 but let's do it later.


SUMMARY:
**Summary:**

`is_torch_compile_available()` checks if the `torch.compile` method is available.

**Key Details:**

* Validates availability of `torch`.
* No version check for native 2.0 compatibility; consider in future iteration.

**Notable Implementation:** 

* `hasattr(torch, "compile")`

Note: relies on `is_torch_available()` which is not shown in this snippet.


key: chroma:document
string_value:
CODE:

def is_torchdynamo_compiling():
    if not is_torch_available():
        return False

    # Importing torch._dynamo causes issues with PyTorch profiler (https://github.com/pytorch/pytorch/issues/130622)
    # hence rather relying on `torch.compiler.is_compiling()` when possible (torch>=2.3)
    try:
        import torch

        return torch.compiler.is_compiling()
    except Exception:
        try:
            import torch._dynamo as dynamo  # noqa: F401

            return dynamo.is_compiling()
        except Exception:


SUMMARY:
**Summary:**

`is_torchdynamo_compiling()` function checks if PyTorch is compiling. The primary purpose is to determine whether PyTorch is in compilation mode.

**Key Details:**

* Checks availability of `torch` and tries alternative imports (`torch.compiler.is_compiling()`) before falling back.
* Imports `torcdynamo` as a last resort, but with a caution due to known compatibility issues ($link).

**Notable Dependencies/Requirements:**

* PyTorch (version >= 2.3)
* External Dependency:
	+ torch (_dynamic) (with noted compatibility issue)


key: chroma:document
string_value:
CODE:

def is_torch_tensorrt_fx_available():
    if importlib.util.find_spec("torch_tensorrt") is None:
        return False


SUMMARY:
**Summary:**

The `is_torch_tensorrt_fx_available` function checks the availability of PyTorch TensorRT FX. It returns `True` if `torch_tensorrt.fx` is found in the Python path, indicating that FX is available for use.

```markdown
### Functionality

* Checks for presence of PyTorch TensorRT FX using `importlib.util.find_spec`
* Returns `True` if `torch_tensorrt.fx` is found; `False` otherwise
```

### Key Implementation Details:

* Tries to find a Python spec for `"torch_tensorrt"` (i.e., the parent package)
* Checks for presence of a sub-module named `"torch_tensorrt.fx"`

### Dependencies and Requirements:

* `importlib.util.find_spec("torch_tensorrt")`
* PyTorch TensorRT FX installed as an external dependency
```


key: chroma:document
string_value:
CODE:

def is_datasets_available():


SUMMARY:
**Summary:**

The `_is_datasets_available` function returns the functionality of the `is_datasets_available` method. It retrieves and returns the value stored in the private variable `__datasets_available`, without modifying it.

**Key Details:**

* Returns the value of `_datasets_available`
* Does not modify the returned value
* Private variable `_datasets_available` is imported from another module (not shown)

**Note:** The function's name `_is_datasets_available` is different from its purpose, which is to return the availability of datasets. Consider renaming the function to `return_dataset_availability`.


key: chroma:document
string_value:
CODE:

def is_detectron2_available():


SUMMARY:
**Summary:**

This is a wrapper function `is_detectron2_available` that returns the result of `_detectron2_available`. The purpose of this function is to provide a interface-agnostic access to a boolean indicating whether Detectron2 is available.



**Key Implementation Details:**

* Returns the value of `_detectron2_available`
* No key algorithm or design choices implemented

**Dependencies/Requirements:**

* `_detectron2_available`: an internal variable or expression, likely referencing a detected installation of Detectron2.
* Access to the implementation of `_detectron2_available` is considered external.


key: chroma:document
string_value:
CODE:

def is_rjieba_available():


SUMMARY:
**Summary:**

The `is_rjieba_available` function checks if the `_rjieba_available` variable is defined. It returns a boolean value indicating whether `_rjieba_available` is True.

**Key Implementation Details:**

* Returns the value of `_rjieba_available`, effectively checking its definition or initialization status.
* Note that this function does not perform any actual check but rather relies on the existence and value of the `_rjieba_available` variable.

**Notable Dependencies/Requirements:**

* `_rjieba_available`: a global or class-level variable (exact type and ownership not specified) used to track availability, with an expected internal representation.


key: chroma:document
string_value:
CODE:

def is_psutil_available():


SUMMARY:
**Summary:**

Function `is_psutil_available()` checks availability of the `_psutil_available` variable, which indicates whether the PS Util library is available.

**Key Points:**

- Returns a boolean value indicating PS Util library availability
- Tied to the pre-defined `_psutil_available` variable


key: chroma:document
string_value:
CODE:

def is_py3nvml_available():


SUMMARY:
**Summary**

This function checks if the `_py3nvml_available` variable is truthy, effectively returning whether `py3nvml` is installed and available.

**Key Details**

* Checks availability of `py3nvml` library
	+ Returns true if `_py3nvml_available` is truthy

**Implementation Note**

The function simply returns the value of `_py3nvml_available`, suggesting that this variable is either set to a boolean indicating availability or initialized with a default value.


key: chroma:document
string_value:
CODE:

def is_sacremoses_available():


SUMMARY:
**Summary:**

`is_sacremoses_available()` Function Summary
=============================================

### Purpose and Functionality

The `is_sacremoses_available()` function checks if the `sacremoses_available` variable is defined and returns its value.

### Key Implementation Details

* The function returns the value of the `_sacremoses_available` variable directly, without any additional processing or filtering.
* No input parameters are accepted.

### Notable Dependencies or Requirements

* `_sacremoses_available`: an external variable reference (its exact definition is not shown in this snippet).

This code snippet appears to be a thin wrapper function, likely used for debugging purposes. The `is_sacremoses_available` function does not perform any meaningful checks or computations, suggesting that the value of `_sacremoses_available` should be directly accessible from elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def is_apex_available():


SUMMARY:
**Summary**

Function `is_apex_available()` checks whether Apex is available. The function simply returns the result of `_apex_available`, implying that it depends on an external variable or state.

**Key Details:**

* Returns the value of `_apex_available`.
* Does not perform any operations to check if Apex is available.
* Relies on an external variable or state (`_apex_available`) for its functionality.


key: chroma:document
string_value:
CODE:

def is_aqlm_available():


SUMMARY:
**Summary:**

### Function Purpose
`is_aqlm_available()` checks whether an internally tracked flag `_aqlm_available` is set to `True`.

### Implementation Details
The function simply returns the value of the `_aqlm_available` variable without any additional logic.

### Notable Dependencies
- `_aqlm_available`: a globally accessible boolean flag.


key: chroma:document
string_value:
CODE:

def is_vptq_available(min_version: str = VPTQ_MIN_VERSION):


SUMMARY:
**Summary**

`is_vptq_available` checks if Virtual Tape Queue (VPTQ) is available based on the minimum required version. It returns `True` if `_vptq_available`, the current version is greater than or equal to the provided minimum version.

**Key Implementation Details:**

* Checks `_vptq_available`
* Verifies version compatibility using `version.parse()`

**Notable Dependencies and Requirements:**

* `_vptq_version`: External dependency (not shown in this snippet)
* `VPTQ_MIN_VERSION` constant
* `version` class or module


key: chroma:document
string_value:
CODE:

def is_av_available():


SUMMARY:
**Summary:**

This function `is_av_available()` returns the availability of an AV (Audio/Video) resource. The implementation delegate this decision to another internal variable `_av_available`. 

**Key Details:**
- Returns bool value indicating AV availability
- Relies on internal `_av_available` attribute
- Simple wrapper without any additional logic

**Dependencies/Requirements:**
- Internal variable/tuple `._av_available` is not shown in this snippet, but presumably a boolean or integer value.


key: chroma:document
string_value:
CODE:

def is_ninja_available():
    r"""
    Code comes from *torch.utils.cpp_extension.is_ninja_available()*. Returns `True` if the
    [ninja](https://ninja-build.org/) build system is available on the system, `False` otherwise.
    """
    try:
        subprocess.check_output("ninja --version".split())
    except Exception:
        return False
    else:


SUMMARY:
**Summary**

This function (`is_ninja_available`) checks if the ninja build system is available on the system.

```markdown
### Function Signature

- `def is_ninja_available():`

### Purpose and Implementation

It attempts to run the `ninja --version` command through `subprocess.check_output()`. If the execution succeeds, it returns `True`; otherwise, it returns `False`.

### Key Implementation Detail

- Runs a version check of the ninja build system using `subprocess.check_output()`.
```

Note: The original comment is preserved but adapted to fit the concise summary format.


key: chroma:document
string_value:
CODE:

def is_ipex_available():
    def get_major_and_minor_from_version(full_version):
        return str(version.parse(full_version).major) + "." + str(version.parse(full_version).minor)

    if not is_torch_available() or not _ipex_available:
        return False

    torch_major_and_minor = get_major_and_minor_from_version(_torch_version)
    ipex_major_and_minor = get_major_and_minor_from_version(_ipex_version)
    if torch_major_and_minor != ipex_major_and_minor:
        logger.warning(
            f"Intel Extension for PyTorch {ipex_major_and_minor} needs to work with PyTorch {ipex_major_and_minor}.*,"
            f" but PyTorch {_torch_version} is found. Please switch to the matching version and run again."
        )
        return False


SUMMARY:
**Function Summary:**

`is_ipex_available()`
================================

Determine if Intel IPEX for PyTorch is available.

### Purpose:

Verify compatibility between PyTorch versions used by Intel IPEX and the required matching version.

### Functionality:

1. Extract major and minor version numbers from `_torch_version` and `_ipex_version`.
2. Check if Torch and IPEX versions match; otherwise, log a warning and return False.
3. If compatible, return True.

### Implementation Details:

* Utilizes `version.parse()` function to extract version numbers.
* Uses `logger.warning()` for compatibility discrepancies.

### Dependencies/Requirements:

* `_torch_available` and `_ipex_available`
* `_torch_version` and `_ipex_version`

### Interface/Methods:

* `_is torch_available()` and `_is_ipex_available()`
* `version.parse()`


key: chroma:document
string_value:
CODE:

def is_bitsandbytes_multi_backend_available() -> bool:
    if not is_bitsandbytes_available():
        return False

    import bitsandbytes as bnb



SUMMARY:
**Summary:**

The `is_bitsandbytes_multi_backend_available` function checks if the "multi_backend" feature is enabled for the BitsAndBytes library. It first verifies that the library is available and then checks its features using attribute access.

**Key Details:**

* Purpose: Verify availability of "multi_backend" feature in BitsAndBytes library
* Implementation:
	+ Imports and initializes BitsAndBytes library (`bitsandbytes as bnb`)
	+ Checks library availability prior to feature verification
* Note: Utilizes `getattr` function to access feature flags within the library

**External Dependencies/Notable Requirements:**

* BitsAndBytes library (`bitsandbytes`)


key: chroma:document
string_value:
CODE:

def is_flash_attn_2_available():
    if not is_torch_available():
        return False

    if not _is_package_available("flash_attn"):
        return False

    # Let's add an extra check to see if cuda is available
    import torch

    if not (torch.cuda.is_available() or is_torch_mlu_available()):
        return False

    if torch.version.cuda:
        return version.parse(importlib.metadata.version("flash_attn")) >= version.parse("2.1.0")
    elif torch.version.hip:
        # TODO: Bump the requirement to 2.1.0 once released in https://github.com/ROCmSoftwarePlatform/flash-attention
        return version.parse(importlib.metadata.version("flash_attn")) >= version.parse("2.0.4")
    elif is_torch_mlu_available():
        return version.parse(importlib.metadata.version("flash_attn")) >= version.parse("2.3.3")
    else:


SUMMARY:
**Summary**

This function checks if a specific version of the `flash_attn` package is available on the system.

**Main Purpose and Functionality**

The function, `is_flash_attn_2_available()`, returns `True` if the required version of `flash_attn` (version 2.1.0 or later) is installed and torch CUDA/MLU is enabled.

**Implementation Details**

* The function first checks for availability of torch and other dependencies.
* It then performs additional checks based on the detected torch CPU/GPU architecture:
	+ For CUDA, it compares the package version with `flash_attn` 2.1.0 or later.
	+ For HIP, it uses an older version threshold until released.
	+ If MLU is available, it uses a different version threshold.

**Notable Dependencies and Requirements**

* `torch`: CPU/GPU availability check
* `_is_package_available("flash_attn")`: package availability check
* `torch.version.cuda` / `torch.version.hip`: CPU/GPU architecture detection
* External dependencies:
	+ torch_mlu (optional)

**Key Method Signatures and Interfaces**

* `version.parse()`: parses version strings from package metadata.


key: chroma:document
string_value:
CODE:

def is_torchdistx_available():


SUMMARY:
**Summary:**

`is_torchdistx_available()` Function Summary:

* **Purpose**: Checks if TorCHDistX is available (`_torchdistx_available` property).
* **Implementation Details**: Directly returns the underlying `_torchdistx_available` value, suggesting a simple wrapper or proxy function.
* **Dependencies/Requirements:** None mentioned explicitly; however, it's assumed that `torchdistx` module is available.

**Class/function name preservation:**

The original code names are preserved: `is_torchdistx_available`, `_torchdistx_available`.

Please let me know if you would like me to further analyze the function or provide an RAG-style summary.


key: chroma:document
string_value:
CODE:

def is_faiss_available():


SUMMARY:
**Summary:**

A read-only wrapper function `_is_faiss_available()` that returns whether the `Faiss` library (not explicitly named in the snippet, but assumed to be an external dependency) is available. The function simply redirects to the original availability check `_faiss_available`. 

**Key Details:**

* Returns a boolean value indicating Availability of 'Faiss' library.

Note: Assuming `_faiss_available` is defined elsewhere and is used to actually check for availability, this wrapper serves as a read-only access layer.


key: chroma:document
string_value:
CODE:

def is_scipy_available():


SUMMARY:
**Summary:**

This Python function `is_scipy_available()` returns a boolean value indicating whether SciPy is available. It simply wraps the internal `_scipy_available` variable, providing a read-only access to its value.

**Key Implementation Details:**

* **Internal Variable:** `_scipy_available`
* **Dependency:** None ( implicit reliance on `_scipy_available` )

**No notable dependencies or external requirements are listed.**


key: chroma:document
string_value:
CODE:

def is_sklearn_available():


SUMMARY:
**Summary:**

`is_sklearn_available()` Function Summary:

* **Purpose**: Determines the availability of scikit-learn library functionality (`_sklearn_available`) for checking if scikit-learn is available.

* Key Details:
	+ Returns whether _sklearn_available is true or false.
	+ Not implementing any additional logic, relying solely on the underlying `_sklearn_available` flag.

* Dependencies/Requirements:
	+ No external dependencies listed.
	+ Assumes the existence of the `_sklearn_available` variable definition in scope. 

Note: This function assumes that `_sklearn_available` is a pre-defined flag or variable indicating the availability of scikit-learn functionality.


key: chroma:document
string_value:
CODE:

def is_sentencepiece_available():


SUMMARY:
**Summary:**

This function `is_sentencepiece_available()` checks if `_sentencepiece_available` attribute is available. It returns the value of the `_sentencepiece_available` attribute directly without any additional logic or processing.

**Key Points:**

* **Purpose:** Checks availability of `_sentencepiece_available` attribute.
* **Function Signature:** `def is_sentencepiece_available(): -> bool = ...`
	+ Returns a boolean indicating whether `_sentencepiece_available` is available.
* **Notable Dependency:** `_sentencepiece_available`

Note: The actual implementation of the `_sentencepiece_available` attribute is not shown in this snippet, but it is assumed to be defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def is_seqio_available():


SUMMARY:
**Summary:**

The `_is_seqio_available` function returns whether the SeqIO library is available, and its wrapper `is_seqio_available()` simply delegates to this internal function.


**Key Implementation Details:**

* `_is_seqio_available` function
	+ Returns a boolean indicating availability of SeqIO
* Wrapping function `is_seqio_available()`
	+ Delegates to `_is_seqio_available`, effectively hiding its implementation details


**Notable Dependencies and Requirements:**

None apparent, as this code segment does not import or reference external dependencies explicitly.


key: chroma:document
string_value:
CODE:

def is_gguf_available(min_version: str = GGUF_MIN_VERSION):


SUMMARY:
**Summary:**

`is_gguf_available` function checks if a specified GGUf plugin version is available.

**Key Implementation Details:**

* Checks the `GGUF_MIN_VERSION` constant and compares it with the parsed `_gguf_version`.
* Returns true if both conditions are met, indicating the GGUf plugin version is available.

**Notable Dependencies:**

* Uses the `_is_gguf_available` variable.
* References `version.parse()` function.
* Implies the use of a `GGUF_MIN_VERSION` constant and an internal `_gguf_version` attribute.


key: chroma:document
string_value:
CODE:

def is_protobuf_available():
    if importlib.util.find_spec("google") is None:
        return False


SUMMARY:
**Summary:**

This Python function checks if the Google Protocol Buffer (protobuf) library is available for import.

* **Functionality:** Returns `True` if protobuf is found, `False` otherwise.
* **Implementation Details:**
	+ Uses `importlib.util.find_spec()` to check for availability of the `google.protobuf` package.
* **Notable Dependencies/Requirements:**
	+ Python's `importlib` module
	+ Google Protocol Buffer library (`google` and `google.protobuf`) (not explicitly imported)
* **Primary Purpose:** Verifies presence of protobuf library, indicating its availability.


key: chroma:document
string_value:
CODE:

def is_fsdp_available(min_version: str = FSDP_MIN_VERSION):


SUMMARY:
**Summary**

`is_fsdp_available` checks if Firebase SDK for Deepfried Plan (FSDP) is available using Torch as a backend. It returns `True` if:

- FSDP and Torch are both installed.
- The minimum required FSDP version (`min_version`) is compatible with the current `_torch_version`.

**Key Details**

* Checks availability of `is_torch_available()` and `_torch_version`.
* Compares `min_version` with parsed version using `version.parse()`.
* Returns result based on the combination of both conditions.

**Notable Dependencies/Requirements**

* Torch
* Firebase SDK (FSDP)
* `version` library for parsing version strings.


key: chroma:document
string_value:
CODE:

def is_optimum_available():


SUMMARY:
**Summary:**

### `is_optimum_available()` Function

**Primary Purpose:** This function returns the result of `_optimum_available`, indicating whether something is optimum available.

**Implementation Details:**

* Returns the value of `_optimum_available` directly, suggesting an intention to keep state outside this scope or a dependency on another module not shown.
* No implementation logic or conditions are provided; only the state or value being read.

**Dependencies/Requirements:**

* `_optimum_available`: A variable or function assumed to be defined elsewhere, possibly in the same or another module.


key: chroma:document
string_value:
CODE:

def is_auto_awq_available():


SUMMARY:
**Summary:**

The `_is_auto_awq_available` function retrieves the value of the `is_auto_awq_available` property, which is stored in the `_auto_awq_available` variable.

**Note:** The actual function name is not used within the code, and it indirectly calls `_auto_awq_available`. For the RAG system, I recommend referencing both `_is_auto_awq_available` (as defined here) and `_auto_awq_available` to maintain consistency.


key: chroma:document
string_value:
CODE:

def is_optimum_quanto_available():
    # `importlib.metadata.version` doesn't work with `optimum.quanto`, need to put `optimum_quanto`


SUMMARY:
**Summary**

`is_optimum_quanto_available()` is a function that checks if version 1.0.0 of the `optimum.quanto` package is available, as `importlib.metadata.version` can't be used for the exact package name.

**Key Details:**

* Function returns boolean indicating availability.
* Calls `_is_optimum_quanto_available()` internal function.

**Notable Dependencies/Interfaces:**

* `optimum.quanto` package (v1.0.0 specific version)
* `importlib.metadata.version` compatibility issue with exact package name
* `_is_optimum_quanto_available()` internal interface not explicitly shown in this code snippet.


key: chroma:document
string_value:
CODE:

def is_compressed_tensors_available():


SUMMARY:
**Summary:**

This is an alias function that returns the availability of compressed tensors. The function re-assigns the `_compressed_tensors_available` protected variable, making it publicly accessible through this API.

**Key Details:**

* Returns: `_compressed_tensors_available`
* Re-assigns a protected variable for public access
* Purpose: Expose tensor compression availability

Note that without the surrounding code, I could not identify any specific implementation details or external dependencies. The provided summary is based on the given function syntax only.


key: chroma:document
string_value:
CODE:

def is_auto_gptq_available():


SUMMARY:
**Summary:**

`is_auto_gptq_available()` Function:

* Purpose: Checks if the AutoGPTQ module is available and returns a boolean value.

**Key Details:**

* Returns the result of `_auto_gptq_available`, indicating its availability status.

**Notable Dependencies:**

* `_auto_gptq_available`
* _ (module not specified in provided context, but possibly referring to 'autogptq' module)


key: chroma:document
string_value:
CODE:

def is_eetq_available():


SUMMARY:
**Summary:**

### Function Purpose:
`is_eetq_available()` returns a boolean indication of whether `_eetq_available` functionality is accessible.

### Implementation Details:

* Returns the result of `_eetq_available`, without wrapping it in any additional logic.
* Does not perform any checks or calculations, simply retrieving the existing value.

### Notable Dependencies:

* `_eetq_available`: a cached boolean indicating availability of EetQ functionality.


key: chroma:document
string_value:
CODE:

def is_fbgemm_gpu_available():


SUMMARY:
**Summary:**

`is_fbgemme_gpu_available()` checks if the FBGEMM GPU availability feature is enabled. Method returns a cached value from `_fbgemm_gpu_available`.

*   **Primary Purpose:** Determines if the FBGEMM GPU feature is available.

*   **Key Implementation Details:**

    *   The method uses a caching mechanism to store and return the availability status of `/_fbgemm_gpu_available`.
    
*   **Notable Dependencies or Requirements:**

    *   `_fbgemm_gpu_available`: A flag indicating whether FBGEMM GPU is available (no direct dependency shown, but assumed).
        External dependencies not explicitly mentioned, assuming a base library provides this flag.


key: chroma:document
string_value:
CODE:

def is_levenshtein_available():


SUMMARY:
**Summary:**

`is_levenshtein_available` Function (Implementation Not Provided)

* `Primary Purpose:` Checks the availability of Levenshtein distance algorithm functionality.
 
No key implementation details are provided, suggesting that the actual Levenshtein distance calculation is delegated to an external function or module.


key: chroma:document
string_value:
CODE:

def is_optimum_neuron_available():


SUMMARY:
**Summary:**

The `is_optimum_neuron_available` function checks if the `_optimum_available` flag is True and if the "optimum.neuron" package is installed. It returns a boolean indicating whether both conditions are met.

**Key Details:**

* Uses private flags `_optimum_available`
* Calls a private method `_is_package_available` to verify package installation
* Package verified for "optimum.neuron"

**No notable dependencies or requirements mentioned.**


key: chroma:document
string_value:
CODE:

def is_safetensors_available():


SUMMARY:
**Summary**

This inline function checks if `is_safetensors_available` is wrapped and calls the underlying `_safetensors_available` function.

**Key Points:**

* `is_safetensors_available`: Checks wrapper existence
* Calls `_safetensors_available`
* Function returns a boolean result


key: chroma:document
string_value:
CODE:

def is_tokenizers_available():


SUMMARY:
**Summary:**

The `is_tokenizers_available` function checks whether the `_tokenizers_available` attribute is accessible, typically indicating whether tokenization capabilities are enabled or available.

**Key Details:**

* Exposes an attribute `_tokenizers_available`
* Return type is boolean (`True` for availability, possibly other values indicated by _tokenizers_available)

This code snippet seems to be a wrapper around an internal attribute, providing a convenient way to check its presence without directly accessing the underlying attribute.


key: chroma:document
string_value:
CODE:

def is_pytesseract_available():


SUMMARY:
**Summary:**

This is a wrapper function that merely returns the result of `_pytesseract_available`. It appears to be part of an authentication mechanism or a flag indicator, but without further context, its primary purpose remains unclear.

**Key Implementation Details:**

* The function uses a private variable (_pytesseract_available) to store the availability status.
* It does not implement any logic for checking the availability; instead, it simplifies this functionality by forwarding the result directly.

**Notable Dependencies or Requirements:**

* `pandas` library is not mentioned explicitly, but given the Python-based context and the mention of _pytesseract_available, it's likely that pytesseract has been imported as a related dependency.
 
The code maintains simplicity while reusing an existing value.


key: chroma:document
string_value:
CODE:

def is_pytest_available():


SUMMARY:
**Summary:**

The `is_pytest_available` function serves as a wrapper around the `_pytest_available` variable, returning its value.

**Key Details:**

* Returns the boolean value of `_pytest_available`

**Implementation Context:**

This function likely belongs to a testing framework or testing utilities namespace.


key: chroma:document
string_value:
CODE:

def is_spacy_available():


SUMMARY:
**Summary:**

This is a wrapper function around the `_spacy_available` variable, which checks if Spacy is available. The primary purpose is to provide a clean interface for determining availability.

**Key Implementation Details:**

* Returns the pre-existing value of `_spacy_available`

**Notable Dependencies and Requirements:**

* _spacy_available (previously defined variable, but not shown in this snippet)


key: chroma:document
string_value:
CODE:

def is_tensorflow_text_available():


SUMMARY:
**Summary:**
  
  Checks if TensorFlow Text is available. Returns `True` only if both TensorFlow and TensorFlow Text are installed.

  **Main Purpose:** Verify presence of TensorFlow Text dependency.

  **Implementation Details:** Uses conditional statement to check availability of `is_tf_available()` and `_tensorflow_text_available`.

  **Key Method Signature:**
    
  - `is_tf_available()`
  - `_tensorflow_text_available`
  
  **Notable Dependencies/Requirements:**
   
  - `TensorFlow` (`is_tf_available`)
  - `TensorFlow Text` (`_tensorflow_text_available`)


key: chroma:document
string_value:
CODE:

def is_keras_nlp_available():


SUMMARY:
**Code Summary**

`is_keras_nlp_available()` function checks if Keras NLP is available by verifying both `tensorflow_text` availability and internal `_keras_nlp_available`.

**Key Insights:**

* Main purpose: Evaluate Keras NLP dependency
* Implementation details:
	+ `is_tensorflow_text_available()` check, suggesting TensorFlow Text integration.
	+ Conditional evaluation of `_keras_nlp_available`, indicating an internal implementation flag for Keras NLP.
* Note: External dependencies not explicitly stated but could be inferred as part of the wider Keras ecosystem.


key: chroma:document
string_value:
CODE:

def is_in_notebook():
    try:
        # Test adapted from tqdm.autonotebook: https://github.com/tqdm/tqdm/blob/master/tqdm/autonotebook.py
        get_ipython = sys.modules["IPython"].get_ipython
        if "IPKernelApp" not in get_ipython().config:
            raise ImportError("console")
        # Removed the lines to include VSCode
        if "DATABRICKS_RUNTIME_VERSION" in os.environ and os.environ["DATABRICKS_RUNTIME_VERSION"] < "11.0":
            # Databricks Runtime 11.0 and above uses IPython kernel by default so it should be compatible with Jupyter notebook
            # https://docs.microsoft.com/en-us/azure/databricks/notebooks/ipython-kernel
            raise ImportError("databricks")

        return importlib.util.find_spec("IPython") is not None
    except (AttributeError, ImportError, KeyError):


SUMMARY:
**Summary: `is_in_notebook()` Function**

**Purpose:** Detects whether the code is being executed within a Jupyter Notebook.

**Functionality:**

- Checks for the presence of IPython kernel (`IPython`) in the current environment.
- Verifies that Databricks Runtime version is 11.0 or above, if present.
- Returns `True` if `IPython` module is imported, indicating Notebook execution. Otherwise, returns `False`.

**Key Implementation Details:**

* Uses `sys.modules["IPython"].get_ipython()` to access IPython instance.
* Expects `DATABRICKS_runtime_VERSION` environment variable to be available (`< 11.0`).

**Dependencies and Requirements:**

* `importlib.util.find_spec("IPython")`
* `os.environ` with `DADATARICS_RUNTIME_VERSION` variable (optional)
* IPython kernel implementation
* Jupyter Notebook environment


key: chroma:document
string_value:
CODE:

def is_pytorch_quantization_available():


SUMMARY:
**Summary:**

The `is_pytorch_quantization_available()` function checks if PyTorch quantization is available. It simply wraps around the `_pytorch_quantization_available` variable, returning its value without any additional validation or processing.

**Key Points:**

* Purpose: Checks availability of PyTorch quantization
* Main Functionality: Returns boolean value depending on `_pytorch_quantization_available`
* Implementation Details: None (simple wrapper function)
* Dependencies:
	+ `_pytorch_quantization_available` variable

Note: The actual implementation and dependencies are determined by the surrounding code, but this summary focuses solely on the provided `is_pytorch_quantization_available()` function.


key: chroma:document
string_value:
CODE:

def is_tensorflow_probability_available():


SUMMARY:
**Summary:**

`is_tensorflow_probability_available()` Function

* **Purpose:** Checks availability of TensorFlow Probability module
* **Main Implementation:** Directly returns the result of `_tensorflow_probability_available`, without additional logic.

**Implementation Details:**

* Does not utilize any specific key algorithms or patterns.
* Utilizes a simple return statement to access the result of `_tensorflow_probability_available`.

**Notable Dependencies/Requirements:**

None explicitly mentioned. However, this function relies on the presence of the `_tensorflow_probability_available` variable, which is not defined in this snippet but seems to be assumed as a global variable or a predefined constant.


key: chroma:document
string_value:
CODE:

def is_pandas_available():


SUMMARY:
**Summary**

```python
is_pandas_available() -> bool:
  """
  Returns a boolean indicating whether pandas library is available.

  Primary purpose: Determine pandas library availability.
  """
  
  # Returns a cached value from the _pandas_available variable.
  return _pandas_available
```

**Key Implementation Details:**

- Caches the result of `_pandas_available` for performance.
- Uses precise technical term "_pandas_available".

**Notable Dependencies/Requirements:**

- `_pandas_available` (local variable).
 
**Interfaces Implemented: None**,


key: chroma:document
string_value:
CODE:

def is_sagemaker_dp_enabled():
    # Get the sagemaker specific env variable.
    sagemaker_params = os.getenv("SM_FRAMEWORK_PARAMS", "{}")
    try:
        # Parse it and check the field "sagemaker_distributed_dataparallel_enabled".
        sagemaker_params = json.loads(sagemaker_params)
        if not sagemaker_params.get("sagemaker_distributed_dataparallel_enabled", False):
            return False
    except json.JSONDecodeError:
        return False
    # Lastly, check if the `smdistributed` module is present.


SUMMARY:
**Summary:**

### Functionality:

The `is_sagemaker_dp_enabled()` function checks whether SageMaker Distributed Parallelism (DP) is enabled on a system. It retrieves the `SM_FRAMEWORK_PARAMS` environment variable, attempts to parse its JSON format, and verifies the presence of the `sagemaker_distributed_dataparallel_enabled` field. If parsing fails or the field is not set to True, the function returns False.

### Key Details:

- Parses `OS Environmental Variable: SM_FRAMEWORK_PARAMS`
- Verifies `JSON-decoded value for sagemaker_distributed_dataparallel_enabled`
  - Returns False on JSON Decode Error

### Dependencies and Requirements:

- `smdistributed_available` (local module)
- `json` library
- (Optional) SageMaker environment with `SM_FRAMEWORK_PARAMS` set


key: chroma:document
string_value:
CODE:

def is_sagemaker_mp_enabled():
    # Get the sagemaker specific mp parameters from smp_options variable.
    smp_options = os.getenv("SM_HP_MP_PARAMETERS", "{}")
    try:
        # Parse it and check the field "partitions" is included, it is required for model parallel.
        smp_options = json.loads(smp_options)
        if "partitions" not in smp_options:
            return False
    except json.JSONDecodeError:
        return False

    # Get the sagemaker specific framework parameters from mpi_options variable.
    mpi_options = os.getenv("SM_FRAMEWORK_PARAMS", "{}")
    try:
        # Parse it and check the field "sagemaker_distributed_dataparallel_enabled".
        mpi_options = json.loads(mpi_options)
        if not mpi_options.get("sagemaker_mpi_enabled", False):
            return False
    except json.JSONDecodeError:
        return False
    # Lastly, check if the `smdistributed` module is present.


SUMMARY:
**Summary:**

### Function Purpose and Functionality:

The `is_sagemaker_mp_enabled()` function checks if SageMaker MP (Model Parallelism) is enabled in the underlying architecture. It verifies two key factors:

1.  Presence of `partitions` field in `smp_options`, indicating model parallel support.
2.  Enablement of the `sagemaker_mpi` framework.

If either condition fails, it returns False.

### Key Implementation Details:

-   Checks for the presence of specific fields using JSON parsing and verification.
-   Uses try-except blocks to handle invalid JSON formats.
-   Calls `_smdistributed_available` (not shown in this code snippet) to ensure module availability.

### Notable Dependencies or Requirements:

-   `os`, `json`, and `_smdistributed` modules are required for environment variable access, JSON parsing, and module checking respectively.

This function ensures compatibility with SageMaker MP by verifying necessary configuration settings and framework enablement.


key: chroma:document
string_value:
CODE:

def is_training_run_on_sagemaker():


SUMMARY:
**Summary**

Checks if a SageMaker job name is present in the environment variables.

**Key Implementation Details**

* Uses `os.environ` to access environmental variables
* Directly checks for presence of `"SAGEMAKER_JOB_NAME"` in the env vars using simple string comparison

**Notable Dependencies or Requirements**

* None apparent, but assumes existence of `os` module as a standard Python dependency.


key: chroma:document
string_value:
CODE:

def is_soundfile_available():


SUMMARY:
**Summary:**

A getter function checking the availability of a `_soundfile_available` flag, returning its current state.

**Key Implementation Details:**

* Primary purpose: Retrieve the value of `_soundfile_available`
* Method signature: `is_soundfile_available() -> bool`

**Notable Dependencies and Requirements:**

* `_soundfile_available`: global variable (value returned by this function)


key: chroma:document
string_value:
CODE:

def is_timm_available():


SUMMARY:
**Summary:**

The `_is_timm_available` function checks if the `TImm` library (`_timm_available`) is available.

* Primary purpose: Returns boolean indicating availability of TImm library
* Key implementation detail: Directly returns the value of `_timm_available`, implying a global or module-scoped variable.
* Notable dependency:
	+ `_timm_available`: Global or module-scoped variable representing the availability status of the TImm library.

Note: The function name is inconsistent with its purpose. It should be renamed to something like `is_timm_enabled` for clarity.


key: chroma:document
string_value:
CODE:

def is_natten_available():


SUMMARY:
**Summary:**

`is_natten_available()` function checks the availability of `_natten_available`. This function directly returns a cached value stored in `_natten_available`, implying it's likely used for memoization or optimization purposes.

**Detailed Analysis:**

- **Purpose:** Checks if `_natten_available` is set to `True`.
- **Key Implementation Details:**
  - Cached result: The function returns the cached value of `_natten_available`.
- **Notable Dependencies:**
  - `_natten_available`: A variable that stores a boolean value indicating availability.


key: chroma:document
string_value:
CODE:

def is_nltk_available():


SUMMARY:
**Summary:**

This is a simple utility function `is_nltk_available()` that returns a boolean value indicating whether Natural Language Toolkit (NLTK) is available. The function wraps the internal `_nltk_available` variable, suggesting it may be part of an internal state or configuration object.

**Key Implementation Details:**

* Returns a boolean value indicating NLTK availability.
* Does not perform any external checks or computations; instead relies on an internal flag (_nltk_available).

**Notable Dependencies or Requirements:**

* None explicitly stated in the code snippet. However, it is likely that NLTK is an external dependency required by the surrounding context.
* Inherited/Implemented interfaces: None mentioned directly in this function.


key: chroma:document
string_value:
CODE:

def is_torchaudio_available():


SUMMARY:
**Summary:**

This function `is_torchaudio_available` checks the availability of the `_torchaudio_available` flag, which indicates whether Torchaudio is installed and ready for use.

**Key Points:**

* **Functionality:** Checks for the availability of a global flag (`_torchaudio_available`)
* **Notable Return Value:** Returns True if Torchaudio is available (i.e., `_torchaudio_available` is set), False otherwise.
* **Implementation:** A simple wrapper function that returns the value of `_torchaudio_available`.


key: chroma:document
string_value:
CODE:

def is_torchao_available():


SUMMARY:
**Summary:**

`is_torchao_available()` Function:

*   **Purpose**: Returns a boolean indicating whether TorchAOA is available.

*   **Implementation Details**:
    *   The function simply returns the value of `_torchao_available`, which suggests that `is_torchao_available` serves as a wrapper or alias.

*   **Notable Dependencies/Requirements**: None explicitly mentioned, but relies on the presence and availability of an `_torchao_available` variable.

*   **Key Method Signature**: `def is_torchao_available(): -> bool`

This function appears to provide a simple API to access the TorchAOA availability status.


key: chroma:document
string_value:
CODE:

def is_speech_available():
    # For now this depends on torchaudio but the exact dependency might evolve in the future.


SUMMARY:
**Summary:**

### Function Purpose:

The `is_speech_available()` function checks if Torchaudio is available for speech functionality.

### Key Implementation Details:

* **Depends on**: `_torchaudio_available` (internal flag indicating availability of Torchaudio)

### Notable Dependencies and Requirements:

* Uses internal `_torchaudio_available` variable, no external dependencies mentioned.

### Primary Purpose Statement:

The primary purpose of this function is to provide a boolean indicator (`True/False`) about the availability of torchaudio for speech functionality.


key: chroma:document
string_value:
CODE:

def is_phonemizer_available():


SUMMARY:
**Code Summary**

This `is_phonemizer_available` function checks the availability of the `_phonemizer_available` constant, which is assumed to be defined in another part of the codebase. The function returns this constant directly without any additional processing.

**Key Analysis Points:**

* Function name: `is_phonemizer_available`
* Primary purpose: Checks if `_phonemizer_available` is a constant.
* Key implementation details:
	+ Returns the value of `_phonemizer_available` directly.
* Notable dependencies or requirements:
	+ `_phonemizer_available` constant assumed to be defined elsewhere.


key: chroma:document
string_value:
CODE:

def is_uroman_available():


SUMMARY:
**Function Summary**

### `is_uroman_available()`

Determines whether the `_uroman_available` flag indicates that URoman is available.

*   Returns: The value of `_uroman_available`
*   Note: This function uses a private variable `_uroman_available`, suggesting it's intended to be internal or controlled.


key: chroma:document
string_value:
CODE:

def torch_only_method(fn):
    def wrapper(*args, **kwargs):
        if not _torch_available:
            raise ImportError(
                "You need to install pytorch to use this method or class, "
                "or activate it with environment variables USE_TORCH=1 and USE_TF=0."
            )
        else:
            return fn(*args, **kwargs)



SUMMARY:
**Summary: Torch Only Method Decorator**

This function creates a decorator (`torch_only_method`) that wraps another function `fn`. The decorator checks if PyTorch is installed and available. If not, it raises an import error with instructions to activate PyTorch. If available, the original function `fn` is executed.

**Key Details:**

* Decorator `torch_only_method` ensures PyTorch installation check
* Raises `ImportError` if PyTorch is not installed or not activated

**Notable Details:**

* Uses `_torch_available` variable to track PyTorch availability


key: chroma:document
string_value:
CODE:

def is_ccl_available():


SUMMARY:
**Summary:**

The `is_ccl_available` function returns the result of the `_is_ccl_available` implementation, suggesting a wrapper or alias function to access private implementation details.

**Key Details:**

* Function signature: `is_ccl_available() -> bool`
* Returns: `_is_ccl_available` (private implementation)
* Notable dependency/requirement: `_is_ccl_available` is an external implementation detail not shown in this code snippet.


key: chroma:document
string_value:
CODE:

def is_sudachi_available():


SUMMARY:
**Summary:**

`is_sudachi_available` checks if the Sudachi API library (`_sudachipy`) is available for use.

**Key Implementation Details:**

- Returns `True` if `_sudachippy_available` is set to `True`.
- No algorithm or design choice is evident in this minimal function.
- Notable method signature:
  - None (only a direct return statement)

**Notable Dependencies/Requirements:**
- External dependency on the Sudachi API library (`_sudachipy`) and its availability checker (`_sudachippy_available`).


key: chroma:document
string_value:
CODE:

def get_sudachi_version():


SUMMARY:
**Function Summary:**

`get_sudachi_version()`: Returns the Sudachi version number from the `_sudachipy_version` attribute.

**Key Implementation Details:**

None, as this function is a simple getter that returns a pre-defined value. 

**Notable Dependencies/Requirements:**

* `sudachipy_version` attribute, not explicitly listed but assumed to be part of an external library or module (`_sudachipy_version`).


key: chroma:document
string_value:
CODE:

def is_sudachi_projection_available():
    if not is_sudachi_available():
        return False

    # NOTE: We require sudachipy>=0.6.8 to use projection option in sudachi_kwargs for the constructor of BertJapaneseTokenizer.
    # - `projection` option is not supported in sudachipy<0.6.8, see https://github.com/WorksApplications/sudachi.rs/issues/230


SUMMARY:
**Code Summary**

Ensures the availability of Sudachi projection feature in `is_sudachi_projection_available()`.

### Main Purpose and Functionality

Checks if Sudachi module version is compatible with projection option, requiring at least v0.6.8.

```python
def is_sudachi_projection_available():
    # ...
```

### Key Implementation Details

*   Uses `sudachi_available()` function to check availability of Sudachi.
*   Compares `sudachipy` version against the threshold version "0.6.8" to enable or disable projection.

```python
if not is_sudachi_available():
    return False

return version.parse(_sudachipy_version) >= version.parse("0.6.8")
```

### Notable Dependencies and Requirements

*   `sudachi` module with at least version `0.6.8` for projection option to be supported.

```python
# NOTE: requires sudachipy>=0.6.8
```


key: chroma:document
string_value:
CODE:

def is_jumanpp_available():


SUMMARY:
**Summary**

The `is_jumanpp_available()` function checks if the `jumanpp` library is installed and available. It returns `True` if both the `rhoknp` package is found using `importlib.util.find_spec` and the `jumanpp` executable is present in the system's PATH.

**Key Implementation Details**

* `importlib.util.find_spec("rhoknp")`: Searches for the `rhoknp` package.
* `shutil.which("jumanpp")`: Checks for the presence of the `jumanpp` executable in the system's PATH.

**External Dependencies and Requirements**

* `importlib`: The Python module for importing libraries.
* `shutil`: The Python module for file system utilities.


key: chroma:document
string_value:
CODE:

def is_cython_available():


SUMMARY:
**Summary:**

The `is_cython_available()` function checks if the Cython foreign function interface ("pyximport") is installed and available for import in Python. It returns `True` if the package is found, indicating that Cython is accessible.

**Key Details:**

* Uses `importlib.util.find_spec()` to search for the "pyximport" package.
* Returns a boolean value indicating availability of "pyximport".

**Notable Dependencies/Requirements:**

* None mentioned in this code snippet.


key: chroma:document
string_value:
CODE:

def is_jieba_available():


SUMMARY:
**Summary**

The `is_jieba_available` function checks the availability of the Jieba library. It directly returns a boolean indicator, `_jieba_available`, indicating whether Jieba is available for use.

**Key Details:**

* Returns a boolean value (`return _jieba_available`)
* Directly references an external variable (`_jieba_available`)


key: chroma:document
string_value:
CODE:

def is_jinja_available():


SUMMARY:
**Summary:**

```python
is_jinja_available()
```

* **Purpose:** Returns whether Jinja templating environment is available.

* **Key Implementation Detail:** Directly returns the `_jinja_available` variable, indicating the availability of the Jinja library without performing any initialization or checks.
* **Notable Dependencies:** None directly referenced; however, assumes the existence of a private variable `_jinja_available`.


key: chroma:document
string_value:
CODE:

def is_mlx_available():


SUMMARY:
**Summary:**

Function `is_mlx_available`: Returns TRUE if `_mlx_available` flag is set, indicating availability of MXL library functionality.

**Key Details:**

* Returns a boolean value (`TRUE` or FALSE)
* Depends on the state of `_mlx_available` flag

**Notable Dependencies/Requirements:**

* None explicitly stated
* Assumed to rely on an external context with `_mlx_available` flag ( possibly a global variable or a class property)


key: chroma:document
string_value:
CODE:

def is_tiktoken_available():


SUMMARY:
**Summary**

This function checks if both `_tiktoken_available` and `_blobfile_available` are True, indicating the availability of TikTok tokens.

**Details**
- Determines the availability of TikTok tokens (`_tiktoken_available`) and blob files (`_blobfile_available`) using logical AND operation.
- Returns a boolean value (`True`/`False`) indicating whether both conditions are met.


key: chroma:document
string_value:
CODE:

def is_liger_kernel_available():
    if not _liger_kernel_available:
        return False



SUMMARY:
**Summary**

Checks the availability and version of LIGER kernel.

**Main Purpose & Functionality**
The `is_liger_kernel_available()` function determines whether the LIGER kernel is available on the system, returning `False` if not found. Otherwise, it checks if the kernel's version meets or exceeds version "0.3.0".

**Key Implementation Details**

* Checks internal `_liger_kernel_available` flag for availability.
* Compares LIGER kernel version with "0.3.0" using `version.parse()` method.

**Notable Dependencies and Requirements**

* Relies on `importlib.metadata.version("liger_kernel")` to retrieve the LIGER kernel's version.
* Requires access to `version.parse()` method for parsing string versions.


key: chroma:document
string_value:
CODE:

def is_triton_available():


SUMMARY:
**Summary:**

This is a wrapper function `is_triton_available` that simply returns the value of another private variable `_triton_available`. The primary purpose is to provide a readable and maintainable interface to access the availability status of a Triton instance.

**Key Implementation Details:**

* Returns a bool indicating Triton availability.
* No actual logic or computation is performed; it merely wraps the value of `_triton_available`.

**Notable Dependencies/Requirements:**

* Access to private variable `_triton_available`.
* Note that Triton is not an external dependency in this code snippet, but its availability status seems to be an ongoing concern.


key: chroma:document
string_value:
CODE:

def requires_backends(obj, backends):
    if not isinstance(backends, (list, tuple)):
        backends = [backends]

    name = obj.__name__ if hasattr(obj, "__name__") else obj.__class__.__name__

    # Raise an error for users who might not realize that classes without "TF" are torch-only
    if "torch" in backends and "tf" not in backends and not is_torch_available() and is_tf_available():
        raise ImportError(PYTORCH_IMPORT_ERROR_WITH_TF.format(name))

    # Raise the inverse error for PyTorch users trying to load TF classes
    if "tf" in backends and "torch" not in backends and is_torch_available() and not is_tf_available():
        raise ImportError(TF_IMPORT_ERROR_WITH_PYTORCH.format(name))

    checks = (BACKENDS_MAPPING[backend] for backend in backends)
    failed = [msg.format(name) for available, msg in checks if not available()]
    if failed:


SUMMARY:
**Function Summary: `requires_backends`**

**Purpose and Functionality**
---------------------------

The `requires_backends` function checks if the provided object requires specific backends for execution, specifically Torch (TF) or not. It verifies compatibility with the user's environment.

**Key Implementation Details**
-----------------------------

* The function raises errors if incompatible backend combinations are used.
* It uses a dictionary (`BACKENDS_MAPPING`) to retrieve checks for each backend.
* Error messages use variable formatting (`PYTORCH_IMPORT_ERROR_WITH_TF` and `TF_IMPORT_ERROR_WITH_PYTORCH`) to include the object's name.

**Notable Dependencies or Requirements**
-----------------------------------------

* `is_torch_available()` and `is_tf_available()` functions, which are not shown in this code snippet.
* External dependencies:
	+ `PYTORCH_IMPORT_ERROR_WITH_TF` and `TF.import_error_with_pytorch` format strings, referencing related classes/functions.


key: chroma:document
string_value:
CODE:

class DummyObject(type):
    """
    Metaclass for the dummy objects. Any class inheriting from it will return the ImportError generated by
    `requires_backend` each time a user tries to access any method of that class.
    """

    def __getattribute__(cls, key):
        if key.startswith("_") and key != "_from_config":
            return super().__getattribute__(key)


SUMMARY:
**Concise Summary**

### Main Purpose and Functionality

The `DummyObject` class is a metaclass that, when used to create a class, renders an `ImportError` exception each time an instance attempts to access any method. This functionality is triggered by the `_getattribute_` method override.

### Implementation Details

* Initializes a class with a required backend configuration using `_backends`
* Catches instance attempts to access methods, returning the required backend configuration
* Prevents direct access to private attributes starting with an underscore

### Notable Dependencies or Requirements

* None external dependencies noted


key: chroma:document
string_value:
CODE:

def is_torch_fx_proxy(x):
    if is_torch_fx_available():
        import torch.fx

        return isinstance(x, torch.fx.Proxy)


SUMMARY:
**Summary:**

`is_torch_fx_proxy(x)` function checks if the input `x` is a Torch FX proxy object. It returns `True` if `x` implements Torch FX's `Proxy` interface and `False` otherwise.

**Key Details:**

* Depends on `torch.fx` module availability
* Uses `isinstance(x, torch.fx.Proxy)` to check for proxy implementation

**Return Value:** Boolean indicating whether `x` is a Torch FX proxy.


key: chroma:document
string_value:
CODE:

class _LazyModule(ModuleType):
    """
    Module class that surfaces all objects but only performs associated imports when the objects are requested.
    """

    # Very heavily inspired by optuna.integration._IntegrationModule
    # https://github.com/optuna/optuna/blob/master/optuna/integration/__init__.py
    def __init__(
        self,
        name: str,
        module_file: str,
        import_structure: IMPORT_STRUCTURE_T,
        module_spec: importlib.machinery.ModuleSpec = None,
        extra_objects: Dict[str, object] = None,
    ):
        super().__init__(name)

        self._object_missing_backend = {}
        if any(isinstance(key, frozenset) for key in import_structure.keys()):
            self._modules = set()
            self._class_to_module = {}
            self.__all__ = []

            _import_structure = {}

            for backends, module in import_structure.items():
                missing_backends = []
                for backend in backends:
                    if backend not in BACKENDS_MAPPING:
                        raise ValueError(
                            f"Error: the following backend: '{backend}' was specified around object {module} but isn't specified in the backends mapping."
                        )
                    callable, error = BACKENDS_MAPPING[backend]
                    if not callable():
                        missing_backends.append(backend)
                self._modules = self._modules.union(set(module.keys()))

                for key, values in module.items():
                    if len(missing_backends):
                        self._object_missing_backend[key] = missing_backends

                    for value in values:
                        self._class_to_module[value] = key
                        if len(missing_backends):
                            self._object_missing_backend[value] = missing_backends
                    _import_structure.setdefault(key, []).extend(values)

                # Needed for autocompletion in an IDE
                self.__all__.extend(list(module.keys()) + list(chain(*module.values())))

            self.__file__ = module_file
            self.__spec__ = module_spec
            self.__path__ = [os.path.dirname(module_file)]
            self._objects = {} if extra_objects is None else extra_objects
            self._name = name
            self._import_structure = _import_structure

        # This can be removed once every exportable object has a `export()` export.
        else:
            self._modules = set(import_structure.keys())
            self._class_to_module = {}
            for key, values in import_structure.items():
                for value in values:
                    self._class_to_module[value] = key
            # Needed for autocompletion in an IDE
            self.__all__ = list(import_structure.keys()) + list(chain(*import_structure.values()))
            self.__file__ = module_file
            self.__spec__ = module_spec
            self.__path__ = [os.path.dirname(module_file)]
            self._objects = {} if extra_objects is None else extra_objects
            self._name = name
            self._import_structure = import_structure

    # Needed for autocompletion in an IDE
    def __dir__(self):
        result = super().__dir__()
        # The elements of self.__all__ that are submodules may or may not be in the dir already, depending on whether
        # they have been accessed or not. So we only add the elements of self.__all__ that are not already in the dir.
        for attr in self.__all__:
            if attr not in result:
                result.append(attr)
        return result

    def __getattr__(self, name: str) -> Any:
        if name in self._objects:
            return self._objects[name]
        if name in self._object_missing_backend.keys():
            missing_backends = self._object_missing_backend[name]

            class Placeholder(metaclass=DummyObject):
                _backends = missing_backends

                def __init__(self, *args, **kwargs):
                    requires_backends(self, missing_backends)

            Placeholder.__name__ = name
            Placeholder.__module__ = self.__spec__

            value = Placeholder
        elif name in self._class_to_module.keys():
            module = self._get_module(self._class_to_module[name])
            value = getattr(module, name)
        elif name in self._modules:
            value = self._get_module(name)
        else:
            raise AttributeError(f"module {self.__name__} has no attribute {name}")

        setattr(self, name, value)
        return value

    def _get_module(self, module_name: str):
        try:
            return importlib.import_module("." + module_name, self.__name__)
        except Exception as e:
            raise RuntimeError(
                f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
                f" traceback):\n{e}"
            ) from e

    def __reduce__(self):


SUMMARY:
**Class Summary**

 `_LazyModule` is a `ModuleType` class that implements lazy loading of objects and modules. It surfaces all objects but only performs associated imports when the objects are requested.

**Key Implementation Details**

*   **Initialization**: Initializes with `name`, `module_file`, `import_structure`, `module_spec`, and `extra_objects`. `import_structure` specifies which backends are required for each imported object.
*   **Object Loading**: Loads missing backends using `__getattr__`. If a backend is missing, it returns a `Placeholder` instance with the required backends. Otherwise, it loads the actual module and attribute values.
*   **Module Retrieval**: Uses `_get_module` to import modules dynamically.

**Notable Dependencies/Requirements**

*   `importlib`: Used for dynamic module importing
*   `BACKENDS_MAPPING`: A dictionary mapping backend instances to their corresponding callable functions

Here's a simple use case:

```python
lazy_module = _LazyModule("my_module", "module.py", {"backends": ["backend1"]})
print(lazy_module.__all__)  # gets the list of objects defined in the module
obj = lazy_module._objects["object"]
setattr(lazy_module, "_attr", obj)
```

Note: The exact implementation and usage might vary based on how you intend to integrate this class. This summary is intended as a starting point for understanding its core functionality.


key: chroma:document
string_value:
CODE:

class OptionalDependencyNotAvailable(BaseException):


SUMMARY:
**Summary:**

 Creates an exception class `OptionalDependencyNotAvailable` ( inherits from `BaseException`) to signal when an optional dependency is not found, intended for internal use. 

**Key Points:**

* Raising an internally used error class
* Signals an absence of an optional dependency
* Inheriting from `BaseException`
* Intended for internal use only


key: chroma:document
string_value:
CODE:

def direct_transformers_import(path: str, file="__init__.py") -> ModuleType:
    """Imports transformers directly

    Args:
        path (`str`): The path to the source file
        file (`str`, *optional*): The file to join with the path. Defaults to "__init__.py".

    Returns:
        `ModuleType`: The resulting imported module
    """
    name = "transformers"
    location = os.path.join(path, file)
    spec = importlib.util.spec_from_file_location(name, location, submodule_search_locations=[path])
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    module = sys.modules[name]


SUMMARY:
**Summary**

Function `direct_transformers_import` imports the Transformers library directly from a given path. It takes two arguments: `path` (source file location) and `file` (the file to join with the path, defaults to "__init__.py"). The function returns the imported module.

**Key Implementation Details**

* Imports `transformers` at specified location using `importlib`.
* Uses `spec_from_file_location` to create a specification for the module.
* Executes the loader of the specification to load the module.
* Returns the loaded module by its name (`sys.modules[name]`).

**Notable Dependencies and Requirements**

* `importlib`: library for dynamic import of modules.
* `os.path`: for joining path with file names.
* `sys.modules`: for accessing imported modules.


key: chroma:document
string_value:
CODE:

def export(*, backends=()):
    """
    This decorator enables two things:
    - Attaching a `__backends` tuple to an object to see what are the necessary backends for it
      to execute correctly without instantiating it
    - The '@export' string is used to dynamically import objects
    """
    for backend in backends:
        if backend not in BACKENDS_MAPPING:
            raise ValueError(f"Backend should be defined in the BACKENDS_MAPPING. Offending backend: {backend}")

    if not isinstance(backends, tuple):
        raise ValueError("Backends should be a tuple.")

    def inner_fn(fun):
        fun.__backends = backends
        return fun



SUMMARY:
**Function Summary**

`export` function is a decorator used to attach necessary backend information and enable dynamic object import. It takes a variable number of `backends` tuples as input, which are validated against the `BACKENDS_MAPPING`. The decorated function (`inner_fn`) stores its associated backends in an internal attribute.

**Key Implementation Details**

- `BACKENDS_MAPPING` dictionary is used for validation and mapping.
- Backends should be a tuple, otherwise a `ValueError` is raised.
- Internal `__backends` attribute is stored in the decorated function to retrieve necessary backends without instantiation.

**Notable Dependencies or Requirements**

- `BACKENDS_MAPPING` class/variable (not shown in this snippet)
- `BACKENDS_MAPPING[backend]` is a valid value


key: chroma:document
string_value:
CODE:

def fetch__all__(file_content):
    """
    Returns the content of the __all__ variable in the file content.
    Returns None if not defined, otherwise returns a list of strings.
    """

    if "__all__" not in file_content:
        return []

    lines = file_content.splitlines()
    for index, line in enumerate(lines):
        if line.startswith("__all__"):
            start_index = index

    lines = lines[start_index:]

    if not lines[0].startswith("__all__"):
        raise ValueError(
            "fetch__all__ accepts a list of lines, with the first line being the __all__ variable declaration"
        )

    # __all__ is defined on a single line
    if lines[0].endswith("]"):
        return [obj.strip("\"' ") for obj in lines[0].split("=")[1].strip(" []").split(",")]

    # __all__ is defined on multiple lines
    else:
        _all = []
        for __all__line_index in range(1, len(lines)):
            if lines[__all__line_index].strip() == "]":
                return _all
            else:
                _all.append(lines[__all__line_index].strip("\"', "))



SUMMARY:
**Function Summary: `fetch__all__(file_content)`**

**Purpose and Functionality:** Retrieves the content of the `__all__` variable from the provided file content.

**Key Implementation Details:**

* Handles single-line and multi-line `__all__` declarations
* Splits and processes `__all__` values into a list of strings

**Notable Dependencies and Requirements:**

* Does not depend on external libraries
* Assumes file content is a string (`file_content`)


key: chroma:document
string_value:
CODE:

def spread_import_structure(nested_import_structure):
    """
    This method takes as input an unordered import structure and brings the required backends at the top-level,
    aggregating modules and objects under their required backends.

    Here's an example of an input import structure at the src.transformers.models level:

    {
        'albert': {
            frozenset(): {
                'configuration_albert': {'AlbertConfig', 'AlbertOnnxConfig'}
            },
            frozenset({'tokenizers'}): {
                'tokenization_albert_fast': {'AlbertTokenizerFast'}
            },
        },
        'align': {
            frozenset(): {
                'configuration_align': {'AlignConfig', 'AlignTextConfig', 'AlignVisionConfig'},
                'processing_align': {'AlignProcessor'}
            },
        },
        'altclip': {
            frozenset(): {
                'configuration_altclip': {'AltCLIPConfig', 'AltCLIPTextConfig', 'AltCLIPVisionConfig'},
                'processing_altclip': {'AltCLIPProcessor'},
            }
        }
    }

    Here's an example of an output import structure at the src.transformers.models level:

    {
        frozenset({'tokenizers'}): {
            'albert.tokenization_albert_fast': {'AlbertTokenizerFast'}
        },
        frozenset(): {
            'albert.configuration_albert': {'AlbertConfig', 'AlbertOnnxConfig'},
            'align.processing_align': {'AlignProcessor'},
            'align.configuration_align': {'AlignConfig', 'AlignTextConfig', 'AlignVisionConfig'},
            'altclip.configuration_altclip': {'AltCLIPConfig', 'AltCLIPTextConfig', 'AltCLIPVisionConfig'},
            'altclip.processing_altclip': {'AltCLIPProcessor'}
        }
    }

    """

    def propagate_frozenset(unordered_import_structure):
        tuple_first_import_structure = {}
        for _key, _value in unordered_import_structure.items():
            if not isinstance(_value, dict):
                tuple_first_import_structure[_key] = _value

            elif any(isinstance(v, frozenset) for v in _value.keys()):
                # Here we want to switch around key and v
                for k, v in _value.items():
                    if isinstance(k, frozenset):
                        if k not in tuple_first_import_structure:
                            tuple_first_import_structure[k] = {}
                        tuple_first_import_structure[k][_key] = v

            else:
                tuple_first_import_structure[_key] = propagate_frozenset(_value)

        return tuple_first_import_structure

    def flatten_dict(_dict, previous_key=None):
        items = []
        for _key, _value in _dict.items():
            _key = f"{previous_key}.{_key}" if previous_key is not None else _key
            if isinstance(_value, dict):
                items.extend(flatten_dict(_value, _key).items())
            else:
                items.append((_key, _value))
        return dict(items)

    # The tuples contain the necessary backends. We want these first, so we propagate them up the
    # import structure.
    ordered_import_structure = nested_import_structure

    # 6 is a number that gives us sufficient depth to go through all files and foreseeable folder depths
    # while not taking too long to parse.
    for i in range(6):
        ordered_import_structure = propagate_frozenset(ordered_import_structure)

    # We then flatten the dict so that it references a module path.
    flattened_import_structure = {}
    for key, value in ordered_import_structure.copy().items():
        if isinstance(key, str):
            del ordered_import_structure[key]
        else:
            flattened_import_structure[key] = flatten_dict(value)



SUMMARY:
**Summary**

`spread_import_structure` is a method that reorganizes an unordered import structure to prioritize required backends at the top-level, aggregating modules and objects under their respective backends. It takes an input nested dictionary with `frozenset` keys and recursively propagates them upwards, flattening the resulting structure to reference module paths.

**Key Insights**

* The method uses two recursive helper functions: `propagate_frozenset` to handle `frozenset` keys and `flatten_dict` to flatten the nested dictionary.
* It prioritizes backend modules at the top-level by propagating them upwards through the import structure using a depth-first approach (`6` iterations).
* The resulting structure references module paths, with backends at the top level and submodules underneath.

**Notable Implementations**

* `ordered_import_structure`: the input nested dictionary with `frozenset` keys.
* `flattened_import_structure`: the output dictionary referencing module paths.

**Dependencies/Interfaces**

* Relies on built-in Python data structures (`dict`, `frozenset`).
* No external dependencies mentioned.


