# embedding_metadata

key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/image_processing_new_imgproc_model.py


key: chroma:document
string_value:
CODE:

class ImgprocModelImageProcessor(BaseImageProcessor):
    r"""
    Constructs a IMGPROC_MODEL image processor.

    Args:
        do_resize (`bool`, *optional*, defaults to `True`):
            Whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by the
            `do_resize` parameter in the `preprocess` method.
        size (`dict`, *optional*, defaults to `{"height": 384, "width": 384}`):
            Size of the output image after resizing. Can be overridden by the `size` parameter in the `preprocess`
            method.
        resample (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`):
            Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. Can be
            overridden by the `resample` parameter in the `preprocess` method.
        do_rescale (`bool`, *optional*, defaults to `True`):
            Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the
            `do_rescale` parameter in the `preprocess` method.
        rescale_factor (`int` or `float`, *optional*, defaults to `1/255`):
            Scale factor to use if rescaling the image. Only has an effect if `do_rescale` is set to `True`. Can be
            overridden by the `rescale_factor` parameter in the `preprocess` method.
        do_normalize (`bool`, *optional*, defaults to `True`):
            Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess`
            method. Can be overridden by the `do_normalize` parameter in the `preprocess` method.
        image_mean (`float` or `List[float]`, *optional*, defaults to `IMAGENET_STANDARD_MEAN`):
            Mean to use if normalizing the image. This is a float or list of floats the length of the number of
            channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be
            overridden by the `image_mean` parameter in the `preprocess` method.
        image_std (`float` or `List[float]`, *optional*, defaults to `IMAGENET_STANDARD_STD`):
            Standard deviation to use if normalizing the image. This is a float or list of floats the length of the
            number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method.
            Can be overridden by the `image_std` parameter in the `preprocess` method.
        do_convert_rgb (`bool`, *optional*, defaults to `True`):
            Whether to convert the image to RGB.
    """

    model_input_names = ["pixel_values"]

    def __init__(
        self,
        do_resize: bool = True,
        size: Dict[str, int] = None,
        resample: PILImageResampling = PILImageResampling.BICUBIC,
        do_rescale: bool = True,
        rescale_factor: Union[int, float] = 1 / 255,
        do_normalize: bool = True,
        image_mean: Optional[Union[float, List[float]]] = None,
        image_std: Optional[Union[float, List[float]]] = None,
        do_convert_rgb: bool = True,
        **kwargs,
    ) -> None:
        super().__init__(**kwargs)
        size = size if size is not None else {"height": 384, "width": 384}
        size = get_size_dict(size, default_to_square=True)

        self.do_resize = do_resize
        self.size = size
        self.resample = resample
        self.do_rescale = do_rescale
        self.rescale_factor = rescale_factor
        self.do_normalize = do_normalize
        self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN
        self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD
        self.do_convert_rgb = do_convert_rgb

    def resize(
        self,
        image: np.ndarray,
        size: Dict[str, int],
        resample: PILImageResampling = PILImageResampling.BICUBIC,
        data_format: Optional[Union[str, ChannelDimension]] = None,
        input_data_format: Optional[Union[str, ChannelDimension]] = None,
        **kwargs,
    ) -> np.ndarray:
        """
        Resize an image to `(size["height"], size["width"])`.

        Args:
            image (`np.ndarray`):
                Image to resize.
            size (`Dict[str, int]`):
                Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image.
            resample (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BICUBIC`):
                `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BICUBIC`.
            data_format (`ChannelDimension` or `str`, *optional*):
                The channel dimension format for the output image. If unset, the channel dimension format of the input
                image is used. Can be one of:
                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
            input_data_format (`ChannelDimension` or `str`, *optional*):
                The channel dimension format for the input image. If unset, the channel dimension format is inferred
                from the input image. Can be one of:
                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.

        Returns:
            `np.ndarray`: The resized image.
        """
        size = get_size_dict(size)
        if "height" not in size or "width" not in size:
            raise ValueError(f"The `size` dictionary must contain the keys `height` and `width`. Got {size.keys()}")
        output_size = (size["height"], size["width"])
        return resize(
            image,
            size=output_size,
            resample=resample,
            data_format=data_format,
            input_data_format=input_data_format,
            **kwargs,
        )

    @filter_out_non_signature_kwargs()
    def preprocess(
        self,
        images: ImageInput,
        do_resize: Optional[bool] = None,
        size: Optional[Dict[str, int]] = None,
        resample: PILImageResampling = None,
        do_rescale: Optional[bool] = None,
        rescale_factor: Optional[float] = None,
        do_normalize: Optional[bool] = None,
        image_mean: Optional[Union[float, List[float]]] = None,
        image_std: Optional[Union[float, List[float]]] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        do_convert_rgb: bool = None,
        data_format: ChannelDimension = ChannelDimension.FIRST,
        input_data_format: Optional[Union[str, ChannelDimension]] = None,
    ) -> PIL.Image.Image:
        """
        Preprocess an image or batch of images.

        Args:
            images (`ImageInput`):
                Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
                passing in images with pixel values between 0 and 1, set `do_rescale=False`.
            do_resize (`bool`, *optional*, defaults to `self.do_resize`):
                Whether to resize the image.
            size (`Dict[str, int]`, *optional*, defaults to `self.size`):
                Controls the size of the image after `resize`. The shortest edge of the image is resized to
                `size["shortest_edge"]` whilst preserving the aspect ratio. If the longest edge of this resized image
                is > `int(size["shortest_edge"] * (1333 / 800))`, then the image is resized again to make the longest
                edge equal to `int(size["shortest_edge"] * (1333 / 800))`.
            resample (`PILImageResampling`, *optional*, defaults to `self.resample`):
                Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`.
            do_rescale (`bool`, *optional*, defaults to `self.do_rescale`):
                Whether to rescale the image values between [0 - 1].
            rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`):
                Rescale factor to rescale the image by if `do_rescale` is set to `True`.
            do_normalize (`bool`, *optional*, defaults to `self.do_normalize`):
                Whether to normalize the image.
            image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`):
                Image mean to normalize the image by if `do_normalize` is set to `True`.
            image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`):
                Image standard deviation to normalize the image by if `do_normalize` is set to `True`.
            do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`):
                Whether to convert the image to RGB.
            return_tensors (`str` or `TensorType`, *optional*):
                The type of tensors to return. Can be one of:
                    - Unset: Return a list of `np.ndarray`.
                    - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.
                    - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.
                    - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.
                    - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.
            data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`):
                The channel dimension format for the output image. Can be one of:
                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
                - Unset: Use the channel dimension format of the input image.
            input_data_format (`ChannelDimension` or `str`, *optional*):
                The channel dimension format for the input image. If unset, the channel dimension format is inferred
                from the input image. Can be one of:
                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
        """
        do_resize = do_resize if do_resize is not None else self.do_resize
        resample = resample if resample is not None else self.resample
        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
        do_normalize = do_normalize if do_normalize is not None else self.do_normalize
        image_mean = image_mean if image_mean is not None else self.image_mean
        image_std = image_std if image_std is not None else self.image_std
        do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb

        size = size if size is not None else self.size
        size = get_size_dict(size, default_to_square=False)

        images = make_list_of_images(images)

        if not valid_images(images):
            raise ValueError(
                "Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, "
                "torch.Tensor, tf.Tensor or jax.ndarray."
            )

        validate_preprocess_arguments(
            do_rescale=do_rescale,
            rescale_factor=rescale_factor,
            do_normalize=do_normalize,
            image_mean=image_mean,
            image_std=image_std,
            do_resize=do_resize,
            size=size,
            resample=resample,
        )
        # PIL RGBA images are converted to RGB
        if do_convert_rgb:
            images = [convert_to_rgb(image) for image in images]

        # All transformations expect numpy arrays.
        images = [to_numpy_array(image) for image in images]

        if do_rescale and is_scaled_image(images[0]):
            logger.warning_once(
                "It looks like you are trying to rescale already rescaled images. If the input"
                " images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
            )

        if input_data_format is None:
            # We assume that all images have the same channel dimension format.
            input_data_format = infer_channel_dimension_format(images[0])

        if do_resize:
            images = [
                self.resize(image=image, size=size, resample=resample, input_data_format=input_data_format)
                for image in images
            ]

        if do_rescale:
            images = [
                self.rescale(image=image, scale=rescale_factor, input_data_format=input_data_format)
                for image in images
            ]

        if do_normalize:
            images = [
                self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
                for image in images
            ]

        images = [
            to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format) for image in images
        ]

        encoded_outputs = BatchFeature(data={"pixel_values": images}, tensor_type=return_tensors)

        return encoded_outputs

    def new_image_processing_method(self, pixel_values: torch.FloatTensor):


SUMMARY:
This is a PyTorch-based image processing class. The class seems to be designed for preprocessing and enhancing images in various ways, such as resizing, scaling, normalizing, and converting them to RGB format.

Here's a high-level overview of the class:

- The class has several attributes that determine how images are processed:
  - `do_rescale`: Whether or not to rescale the pixel values.
  - `rescale_factor`: The factor by which the pixel values should be scaled if rescaling is needed.
  - `do_normalize`: Whether or not to normalize the pixel values.
  - `image_mean` and `image_std`: Mean and standard deviation used for normalizing.
  - `size`: The size at which images are resized (for downsampling).
  - `resample`: The method used for resizing (e.g., nearest-neighbor, bicubic).
- It also attributes and methods that handle errors and edge cases:
  - `make_list_of_images` takes a collection of images as input.
  - `validate_preprocess_arguments` checks if all preprocessings arguments are valid.
  - It checks for invalid image type (i.e., PIL Image, np.ndarray, torch.Tensor, tf.Tensor or jax.ndarray).
- The class processes preprocessed images by:
  - Converting them to numpy arrays
  - Rescaling and normalizing these arrays according to their respective settings.
  - After preprocessing, it encodes the output in a format that can be consumed by a certain model.

Here's a summary of what this code does:

* Preprocesses input image(s) based on specified transformation settings
* May rescale pixel values if necessary
* Normalize pixel values with mean and standard deviation (if applicable)
* Apply transformations like resizing or resampling if required

Please provide more context about where you'd like to improve this implementation, to give a clearer idea of what changes are acceptable and relevant.


key: chroma:document
string_value:
CODE:

class DummyRMSNorm(nn.Module):
    def __init__(self, hidden_size, eps=1e-6):
        """
        DummyRMSNorm is equivalent to T5LayerNorm
        """
        super().__init__()
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.variance_epsilon = eps

    def forward(self, hidden_states):
        input_dtype = hidden_states.dtype
        hidden_states = hidden_states.to(torch.float32)
        variance = hidden_states.pow(2).mean(-1, keepdim=True)
        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
        return self.weight * hidden_states.to(input_dtype)

    def extra_repr(self):


SUMMARY:
**Summary: DummyRMSNorm Module**

The `DummyRMSNorm` class is a PyTorch module equivalent to the T5LayerNorm. It normalizes input tensors with a scaling factor determined by the variance of the input.

**Key Implementation Details:**

* Scales input tensor by square root of mean-variance, modified by epsilon value.
* Uses self-weight parameter to scale output.
* Employs `pow` and `mean` operations over input tensor with shape `(batch_size, seq_len, hidden_size)`.

**Notable Dependencies/Requirements:**

* PyTorch library (`torch`, `nn.Module`)
* TensorFlow-compatible data types (`float32`)


key: chroma:document
string_value:
CODE:

class DummyRotaryEmbedding(nn.Module):
    def __init__(
        self,
        config: DummyConfig,
        device=None,
    ):
        super().__init__()
        self.rope_kwargs = {}
        # BC: "rope_type" was originally "type"
        if hasattr(config, "rope_scaling") and config.rope_scaling is not None:
            self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
        else:
            self.rope_type = "default"
        self.max_seq_len_cached = config.max_position_embeddings
        self.original_max_seq_len = config.max_position_embeddings

        self.config = config
        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]

        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device, **self.rope_kwargs)
        self.register_buffer("inv_freq", inv_freq, persistent=False)
        self.original_inv_freq = self.inv_freq

    def _dynamic_frequency_update(self, position_ids, device):
        """
        dynamic RoPE layers should recompute `inv_freq` in the following situations:
        1 - growing beyond the cached sequence length (allow scaling)
        2 - the current sequence length is in the original scale (avoid losing precision with small sequences)
        """
        seq_len = torch.max(position_ids) + 1
        if seq_len > self.max_seq_len_cached:  # growth
            inv_freq, self.attention_scaling = self.rope_init_fn(
                self.config, device, seq_len=seq_len, **self.rope_kwargs
            )
            self.register_buffer("inv_freq", inv_freq, persistent=False)  # TODO joao: may break with compilation
            self.max_seq_len_cached = seq_len

        if seq_len < self.original_max_seq_len and self.max_seq_len_cached > self.original_max_seq_len:  # reset
            self.register_buffer("inv_freq", self.original_inv_freq, persistent=False)
            self.max_seq_len_cached = self.original_max_seq_len

    @torch.no_grad()
    def forward(self, x, position_ids):
        if "dynamic" in self.rope_type:
            self._dynamic_frequency_update(position_ids, device=x.device)

        # Core RoPE block
        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
        position_ids_expanded = position_ids[:, None, :].float()
        # Force float32 (see https://github.com/huggingface/transformers/pull/29285)
        device_type = x.device.type
        device_type = device_type if isinstance(device_type, str) and device_type != "mps" else "cpu"
        with torch.autocast(device_type=device_type, enabled=False):
            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
            emb = torch.cat((freqs, freqs), dim=-1)
            cos = emb.cos()
            sin = emb.sin()

        # Advanced RoPE types (e.g. yarn) apply a post-processing scaling factor, equivalent to scaling attention
        cos = cos * self.attention_scaling
        sin = sin * self.attention_scaling



SUMMARY:
**Summary:**

The `DummyRotaryEmbedding` class is a PyTorch module implementing RoPE (Rotation Embeddings) layers. It adapts to dynamic sequence lengths and applies attention scaling.

**Main Purpose and Functionality:**

This class is designed to compute frequency-based embeddings for input sequences, allowing the model to handle varying sequence lengths efficiently. The main functionality includes:

* Adaptive initialization of `inv_freq` based on sequence length
* Dynamic re-computation of `inv_freq` when sequence length grows or shrinks
* Application of attention scaling factor to post-process embeddings

**Key Implementation Details:**

* Use of `torch.autocast` for dynamic torch device type management
* Custom RoPE initialization function selection based on `rope_type`
* Usage of `position_ids` for frequency-based embedding computation
* Attention scaling factor application for post-processing embeddings

**Notable Dependencies and Requirements:**

* PyTorch module (`nn.Module`)
* External dependencies:
	+ `DummyConfig` class (config passing)
	+ `ROPE_INIT_FUNCTIONS` dictionary (rope type initialization function selection)

**Additional Notes:**

The RoPE implementation utilizes frequency-based embeddings for input sequences, enabling the model to adapt to varying sequence lengths while maintaining efficient computational complexity.


key: chroma:document
string_value:
CODE:

class DummyMLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))


SUMMARY:
**Class Summary: `DummyMLP`**

**Main Purpose and Functionality:** 
The `DummyMLP` class implements a basic multi-layer perceptron (MLP) architecture. It takes in input data `x`, applies actuarial function, combines with two projections (`up_proj` and `down_proj`) and outputs the result.

**Key Implementation Details:**

* Uses feed-forward neural network (`nn.Module`) for model construction.
* Inclodes multiple linear transformations:
	+ `gate_proj`: projects input onto hidden layer
	+ `up_proj`: up-projects output from gateProj
  - `down_proj`: Down-projects and combines with act_fn(x)
* Utilizes an acturial function (`ACT2FN[config.hidden_act]`) to apply activation mapping.

**Notable Dependencies or Requirements:**

* Uses PyTorch's neural network module (`nn.Module`)
* References external dictionary (`ACT2FN`)

Please let me know if you want me to clarify any details.


key: chroma:document
string_value:
CODE:

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 4]
    x2 = x[..., x.shape[-1] // 4 :]


SUMMARY:
**Summary**

Rotates the specified half of the hidden dimensions in input `x`. Specifically, it:

* Splits the input into two halves using index division by 4
* Rotates the second half (`x2`) to the beginning and keeps the first half (`x1`)
* Concatenates the rotated halves with opposite signs using `torch cat`

**Implementation Details**

* Uses PyTorch tensor indexing and concatenation

**Dependencies/Requirements**

* Requires `torch` library for tensor operations.


key: chroma:document
string_value:
CODE:

def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
    """Applies Rotary Position Embedding to the query and key tensors.

    Args:
        q (`torch.Tensor`): The query tensor.
        k (`torch.Tensor`): The key tensor.
        cos (`torch.Tensor`): The cosine part of the rotary embedding.
        sin (`torch.Tensor`): The sine part of the rotary embedding.
        position_ids (`torch.Tensor`, *optional*):
            Deprecated and unused.
        unsqueeze_dim (`int`, *optional*, defaults to 1):
            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
    Returns:
        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
    """
    cos = cos.unsqueeze(unsqueeze_dim)
    sin = sin.unsqueeze(unsqueeze_dim)
    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)


SUMMARY:
**Rotary Position Embedding Function Summary**

### Purpose and Functionality
Applies Rotary Position Embedding to query and key tensors.

#### Key Implementation Details

* Applies Rotatory Position Embedding using `cos(x) + rotate_half(sin(x))`
* Uses cosine (`cos`) and sine (`sin`) components of the embedding as specified arguments.
* Unsqueeze dimensions of `cos` and `sin` tensors for proper broadcasting with query (`q`) and key (`k`) tensors.

#### Notable Dependencies

* Inputs:
	+ `q`: Query tensor (torch.Tensor)
	+ `k`: Key tensor (torch.Tensor)
	+ `cos`, `sin`: Cosine and sine components of the embedding (torch.Tensor)
	+ `position_ids` (Deprecated, Unused): Optional position ID tensor
	+ `unsqueeze_dim`: Dimension to unsqueeze `cos[position_ids]` and `sin[position_ids]` for broadcasting
* Returns: Tuples of rotated query (`q_embed`) and key (`k_embed`) tensors.

#### Dependencies and Interfaces

* No external dependencies.
* Implies use of `torch.Tensor` type and vector operations.


key: chroma:document
string_value:
CODE:

def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
    """
    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
    """
    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
    if n_rep == 1:
        return hidden_states
    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)


SUMMARY:
**Summary:**

The `repeat_kv` function is equivalent to `torch.repeat_interleave`, repeating the `hidden_states` tensor along the key-value heads dimension (`dim=1`) with `n_rep` repetitions. It transforms the input from `(batch, num_key_value_heads, seqlen, head_dim)` to `(batch, num_attention_heads, seqlen, head_dim)`. The function is designed to handle multiple attention heads by repeating each head `n_rep` times.

**Key details:**

* Transform: Repeats along key-value heads dimension (`dim=1`)
* Repeats: `n_rep` repetitions
* Output shape: `(batch, num_attention_heads, seqlen, head_dim)`
* Equivalent to: `torch.repeat_interleave(x, dim=1, repeats=n_rep)`

**Notable dependencies/requirements:**

* `torch`: Required for tensor operations and repeat functionality.


key: chroma:document
string_value:
CODE:

def eager_attention_forward(
    module: nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    scaling: float,
    dropout: float = 0.0,
    **kwargs,
):
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)

    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
    if attention_mask is not None:
        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
        attn_weights = attn_weights + causal_mask

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
    attn_output = torch.matmul(attn_weights, value_states)
    attn_output = attn_output.transpose(1, 2).contiguous()



SUMMARY:
**Concise Summary**

The `eager_attention_forward` function implements multi-head attention mechanism using eager attention. It takes in query, key, value tensors and an optional attention mask. The function:

1. Scales the key-value pairs by a factor of 2^module.num_key_value_groups.
2. Computes attention weights through matrix multiplication between query and scaled keys (dim = -1).
3. Applies causal masking if provided.
4. Softmax normalization and dropout are applied to the attention weights.
5. Computes output by multiplying scaled weights with value tensor.
6. Returns the output and scaled weights.

**Key Implementation Details**

* `torch.matmul` for matrix multiplication
* `nn.functional.softmax` for softmax normalization
* `nn.functional.dropout` for scaling weights

**Dependences and Notes**

* Inheritance: no external modules, only PyTorch's `nn.Module`
* Interfaces: implements Multi-Head Attention (uses repeated key-value groups)
* Scaling factor can be adjusted to the `module.num_key_value_groups`


key: chroma:document
string_value:
CODE:

class DummyAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(self, config: DummyConfig, layer_idx: int):
        super().__init__()
        self.config = config
        self.layer_idx = layer_idx
        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
        self.scaling = self.head_dim**-0.5
        self.attention_dropout = config.attention_dropout
        self.is_causal = True

        self.q_proj = nn.Linear(
            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
        )
        self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
        self.v_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
        self.o_proj = nn.Linear(
            config.num_attention_heads * self.head_dim, config.hidden_size, bias=config.attention_bias
        )

    def forward(
        self,
        hidden_states: torch.Tensor,
        position_embeddings: Tuple[torch.Tensor, torch.Tensor],
        attention_mask: Optional[torch.Tensor],
        past_key_value: Optional[Cache] = None,
        cache_position: Optional[torch.LongTensor] = None,
        **kwargs: Unpack[FlashAttentionKwargs],
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
        input_shape = hidden_states.shape[:-1]
        hidden_shape = (*input_shape, -1, self.head_dim)

        query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)

        cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)

        if past_key_value is not None:
            # sin and cos are specific to RoPE models; cache_position needed for the static cache
            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)

        attention_interface: Callable = eager_attention_forward
        if self.config._attn_implementation != "eager":
            if self.config._attn_implementation == "sdpa" and kwargs.get("output_attentions", False):
                logger.warning_once(
                    "`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to "
                    'eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.'
                )
            else:
                attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]

        attn_output, attn_weights = attention_interface(
            self,
            query_states,
            key_states,
            value_states,
            attention_mask,
            dropout=0.0 if not self.training else self.attention_dropout,
            scaling=self.scaling,
            **kwargs,
        )

        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)


SUMMARY:
**Summary:**

The `DummyAttention` class implements multi-headed attention from the "Attention Is All You Need" paper. It extends PyTorch's `nn.Module` and is used in transformer architectures.

**Key Features:**

* Takes a configuration object, layer index, and input states (hidden_states) as inputs.
* Projects hidden_states to query, key, and value vectors using linear layers.
* Applies position embeddings for cosine and sine values.

**Implementation Details:**

* Uses the `eager` attention implementation by default or based on the `_attn_implementation` configuration.
* Incorporates a past_key_value cache for efficient updates when caching positional information.
* Projects outputs through an output projection layer (`o_proj`) to obtain attention scores.

**Notable Dependencies and Requirements:**

* PyTorch's `nn.Module`
* `dummy_config` class (config object)
* `torch.nn.Linear`, `torch.utils.data.Tensor`


key: chroma:document
string_value:
CODE:

class DummyDecoderLayer(nn.Module):
    def __init__(self, config: DummyConfig, layer_idx: int):
        super().__init__()
        self.hidden_size = config.hidden_size

        self.self_attn = DummyAttention(config=config, layer_idx=layer_idx)

        self.mlp = DummyMLP(config)
        self.input_layernorm = DummyRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
        self.post_attention_layernorm = DummyRMSNorm(config.hidden_size, eps=config.rms_norm_eps)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_value: Optional[Cache] = None,
        output_attentions: Optional[bool] = False,
        use_cache: Optional[bool] = False,
        cache_position: Optional[torch.LongTensor] = None,
        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
        **kwargs: Unpack[FlashAttentionKwargs],
    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            cache_position=cache_position,
            position_embeddings=position_embeddings,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

        outputs = (hidden_states,)
        if output_attentions:
            outputs += (self_attn_weights,)



SUMMARY:
**DummyDecoderLayer Class Summary**

### Purpose and Functionality
The `DummyDecoderLayer` class implements a layer in a transformer decoder architecture. Its primary purpose is to perform self-attention and fully connected linear transformations on input embeddings.

### Key Implementation Details

* Performs self-attention using the `self_attn` method, which takes input embeddings, attention mask, and other optional arguments.
	+ Method Signature: `self.self_attn(hidden_states=..., attention_mask=..., ...)`
	+ Returns: `(hidden_states, self_attn_weights)`
* Applies linear transformation using the `mlp` method, which takes input embeddings after self-attention.
	+ Method Signature: `self.mlp(hidden_states=...)`
	+ Returns: `(hidden_states)`
* Normalizes inputs with layer normalization (`input_layernorm`) and post-attention normalization (`post_attention_layernorm`).

### Notable Dependencies and Requirements

* `config`: an instance of `DummyConfig`, which contains hyperparameters for the layer.
* External dependencies:
	+ `torch`
	+ `nn.Module` (from PyTorch)
	+ `torch.Tensor`
	+ `Optional`
	+ `torch.LongTensor`

### Inherited or Implemented Interfaces

* Implements the `nn.Module` interface from PyTorch.

Note: This summary focuses on the key implementation details and dependencies specific to the `DummyDecoderLayer` class, without providing unnecessary or generic information.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_new_imgproc_model.py


key: chroma:document
string_value:
CODE:

class ImgprocModelImageProcessor(BlipImageProcessor):
    def new_image_processing_method(self, pixel_values: torch.FloatTensor):


SUMMARY:
**Summary:**

This class, `ImgprocModelImageProcessor`, inherits from `BlipImageProcessor`. It defines a method `new_image_processing_method` which takes a `torch.FloatTensor` input `pixel_values` and returns the pixel values divided by 2. 

**Key Details:**

- Algorithm: Simple Linear Scaling
- Method Signature: `torch.FloatTensor -> torch.FloatTensor`
- Interfaces: Inherited from `BlipImageProcessor`

**Notable Dependencies/Requirements:** 
- `torch` tensor library


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/configuration_my_new_model.py


key: chroma:document
string_value:
CODE:

class MyNewModelConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`MyNewModelModel`]. It is used to instantiate an MyNewModel
    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
    defaults will yield a similar configuration to that of the MyNewModel-7B.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        vocab_size (`int`, *optional*, defaults to 32000):
            Vocabulary size of the MyNewModel model. Defines the number of different tokens that can be represented by the
            `inputs_ids` passed when calling [`MyNewModelModel`]
        hidden_size (`int`, *optional*, defaults to 4096):
            Dimension of the hidden representations.
        intermediate_size (`int`, *optional*, defaults to 11008):
            Dimension of the MLP representations.
        num_hidden_layers (`int`, *optional*, defaults to 32):
            Number of hidden layers in the Transformer decoder.
        num_attention_heads (`int`, *optional*, defaults to 32):
            Number of attention heads for each attention layer in the Transformer decoder.
        num_key_value_heads (`int`, *optional*):
            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
            by meanpooling all the original heads within that group. For more details checkout [this
            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
            `num_attention_heads`.
        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
            The non-linear activation function (function or string) in the decoder.
        max_position_embeddings (`int`, *optional*, defaults to 2048):
            The maximum sequence length that this model might ever be used with. MyNewModel 1 supports up to 2048 tokens,
            MyNewModel 2 up to 4096, CodeLlama up to 16384.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
            The epsilon used by the rms normalization layers.
        use_cache (`bool`, *optional*, defaults to `True`):
            Whether or not the model should return the last key/values attentions (not used by all models). Only
            relevant if `config.is_decoder=True`.
        pad_token_id (`int`, *optional*):
            Padding token id.
        bos_token_id (`int`, *optional*, defaults to 1):
            Beginning of stream token id.
        eos_token_id (`int`, *optional*, defaults to 2):
            End of stream token id.
        pretraining_tp (`int`, *optional*, defaults to 1):
            Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
            document](https://huggingface.co/docs/transformers/main/perf_train_gpu_many#tensor-parallelism) to
            understand more about it. This value is necessary to ensure exact reproducibility of the pretraining
            results. Please refer to [this issue](https://github.com/pytorch/pytorch/issues/76232).
        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
            Whether to tie weight embeddings
        rope_theta (`float`, *optional*, defaults to 10000.0):
            The base period of the RoPE embeddings.
        rope_scaling (`Dict`, *optional*):
            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
            accordingly.
            Expected contents:
                `rope_type` (`str`):
                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
                    'my_new_model3'], with 'default' being the original RoPE implementation.
                `factor` (`float`, *optional*):
                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *
                    original maximum pre-trained length.
                `original_max_position_embeddings` (`int`, *optional*):
                    Used with 'dynamic', 'longrope' and 'my_new_model3'. The original max position embeddings used during
                    pretraining.
                `attention_factor` (`float`, *optional*):
                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
                    computation. If unspecified, it defaults to value recommended by the implementation, using the
                    `factor` field to infer the suggested value.
                `beta_fast` (`float`, *optional*):
                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
                    ramp function. If unspecified, it defaults to 32.
                `beta_slow` (`float`, *optional*):
                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
                    ramp function. If unspecified, it defaults to 1.
                `short_factor` (`List[float]`, *optional*):
                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<
                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
                    size divided by the number of attention heads divided by 2
                `long_factor` (`List[float]`, *optional*):
                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<
                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
                    size divided by the number of attention heads divided by 2
                `low_freq_factor` (`float`, *optional*):
                    Only used with 'my_new_model3'. Scaling factor applied to low frequency components of the RoPE
                `high_freq_factor` (`float`, *optional*):
                    Only used with 'my_new_model3'. Scaling factor applied to high frequency components of the RoPE
        attention_bias (`bool`, *optional*, defaults to `False`):
            Whether to use a bias in the query, key, value and output projection layers during self-attention.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities.
        mlp_bias (`bool`, *optional*, defaults to `False`):
            Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers.
        head_dim (`int`, *optional*):
            The attention head dimension. If None, it will default to hidden_size // num_attention_heads

    ```python
    >>> from transformers import MyNewModelModel, MyNewModelConfig

    >>> # Initializing a MyNewModel my_new_model-7b style configuration
    >>> configuration = MyNewModelConfig()

    >>> # Initializing a model from the my_new_model-7b style configuration
    >>> model = MyNewModelModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config
    ```
        new_param (`int`, *optional*, defaults to `False`):
            A fun new parameter
    """

    model_type = "my_new_model"
    keys_to_ignore_at_inference = ["past_key_values"]
    # Default tensor parallel plan for base model `MyNewModelModel`
    base_model_tp_plan = {
        "layers.*.self_attn.q_proj": "colwise",
        "layers.*.self_attn.k_proj": "colwise",
        "layers.*.self_attn.v_proj": "colwise",
        "layers.*.self_attn.o_proj": "rowwise",
        "layers.*.mlp.gate_proj": "colwise",
        "layers.*.mlp.up_proj": "colwise",
        "layers.*.mlp.down_proj": "rowwise",
    }

    def __init__(
        self,
        vocab_size=32000,
        hidden_size=4096,
        intermediate_size=11008,
        num_hidden_layers=32,
        num_attention_heads=32,
        num_key_value_heads=None,
        hidden_act="silu",
        max_position_embeddings=2048,
        initializer_range=0.02,
        rms_norm_eps=1e-6,
        use_cache=True,
        pad_token_id=None,
        bos_token_id=1,
        eos_token_id=2,
        pretraining_tp=1,
        tie_word_embeddings=False,
        rope_theta=10000.0,
        rope_scaling=None,
        attention_bias=False,
        attention_dropout=0.0,
        mlp_bias=True,
        head_dim=None,
        new_param=0,
        **kwargs,
    ):
        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads

        # for backward compatibility
        if num_key_value_heads is None:
            num_key_value_heads = num_attention_heads

        self.num_key_value_heads = num_key_value_heads
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.rms_norm_eps = rms_norm_eps
        self.pretraining_tp = pretraining_tp
        self.use_cache = use_cache
        self.rope_theta = rope_theta
        self.rope_scaling = rope_scaling
        self.attention_bias = attention_bias
        self.attention_dropout = attention_dropout
        self.mlp_bias = mlp_bias
        self.head_dim = head_dim if head_dim is not None else self.hidden_size // self.num_attention_heads
        # Validate the correctness of rotary position embeddings parameters
        # BC: if there is a 'type' field, copy it it to 'rope_type'.
        if self.rope_scaling is not None and "type" in self.rope_scaling:
            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
        rope_config_validation(self)


SUMMARY:
This code defines the configuration for a custom transformer model called `MyNewModel`. It consists of various hyperparameters and settings, such as:

* Model architecture (number of hidden layers, attention heads, intermediate size)
* Training settings (vocab size, max position embeddings, initial optimizer range)
* Hyperparameters (hidden act, beta test parameter)
* Performance metrics and validation scheme
* Output model type, key fields to ignore at inference, default tensor parallel plan

The code is intended for use with a transformer-based machine learning model, where the configuration defines how the model should be trained, processed, and evaluated. 

It's often used in deep learning models such as those in NLP tasks like language translation or question-answering, although it is just a placeholder example of many more transformer based architectures that are built with the Hugging Face framework.

Overall this code provides an object-oriented definition for how transformers should be constructed and configured to handle many aspects of their training and usage.


key: chroma:document
string_value:
CODE:

class DummyBertEmbeddings(nn.Module):
    """Construct the embeddings from word, position and token_type embeddings."""

    def __init__(self, config):
        super().__init__()
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)

        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
        # any TensorFlow checkpoint file
        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        # position_ids (1, len position emb) is contiguous in memory and exported when serialized
        self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")
        self.register_buffer(
            "position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)), persistent=False
        )
        self.register_buffer(
            "token_type_ids", torch.zeros(self.position_ids.size(), dtype=torch.long), persistent=False
        )

    def forward(
        self,
        input_ids: Optional[torch.LongTensor] = None,
        token_type_ids: Optional[torch.LongTensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        inputs_embeds: Optional[torch.FloatTensor] = None,
        past_key_values_length: int = 0,
    ) -> torch.Tensor:
        if input_ids is not None:
            input_shape = input_ids.size()
        else:
            input_shape = inputs_embeds.size()[:-1]

        seq_length = input_shape[1]

        if position_ids is None:
            position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]

        # Setting the token_type_ids to the registered buffer in constructor where it is all zeros, which usually occurs
        # when its auto-generated, registered buffer helps users when tracing the model without passing token_type_ids, solves
        # issue #5664
        if token_type_ids is None:
            if hasattr(self, "token_type_ids"):
                buffered_token_type_ids = self.token_type_ids[:, :seq_length]
                buffered_token_type_ids_expanded = buffered_token_type_ids.expand(input_shape[0], seq_length)
                token_type_ids = buffered_token_type_ids_expanded
            else:
                token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)

        if inputs_embeds is None:
            inputs_embeds = self.word_embeddings(input_ids)
        token_type_embeddings = self.token_type_embeddings(token_type_ids)

        embeddings = inputs_embeds + token_type_embeddings
        if self.position_embedding_type == "absolute":
            position_embeddings = self.position_embeddings(position_ids)
            embeddings += position_embeddings
        embeddings = self.LayerNorm(embeddings)
        embeddings = self.dropout(embeddings)


SUMMARY:
**Summary:**
The `DummyBertEmbeddings` class is a PyTorch module responsible for constructing BERT-style embeddings from word, position, and token type inputs. It extends the PyTorch `nn.Module` class.

**Purpose:** The main purpose of this class is to compute contextualized character-level representations by combining word embeddings with position and token type information.

**Key Implementations:**
* Embeds words using a learned embedding matrix (`word_embeddings`)
* Apps `position_embeddings` from word embeddings based on `position_ids`
* Applies `token_type_embeddings` using an embedding layer for token types
* Employs multi-head attention mechanisms through the `LayerNorm` and `dropout` layers

**Notable Dependencies:**
* PyTorch library (specifically, `torch.nn`)
* The Hugging Face Transformers library is referenced indirectly through its usage of `config` objects


key: chroma:document
string_value:
CODE:

class DummyBertSelfAttention(nn.Module):
    def __init__(self, config, position_embedding_type=None):
        super().__init__()
        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
            raise ValueError(
                f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
                f"heads ({config.num_attention_heads})"
            )

        self.num_attention_heads = config.num_attention_heads
        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
        self.all_head_size = self.num_attention_heads * self.attention_head_size

        self.query = nn.Linear(config.hidden_size, self.all_head_size)
        self.key = nn.Linear(config.hidden_size, self.all_head_size)
        self.value = nn.Linear(config.hidden_size, self.all_head_size)

        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
        self.position_embedding_type = position_embedding_type or getattr(
            config, "position_embedding_type", "absolute"
        )
        if self.position_embedding_type == "relative_key" or self.position_embedding_type == "relative_key_query":
            self.max_position_embeddings = config.max_position_embeddings
            self.distance_embedding = nn.Embedding(2 * config.max_position_embeddings - 1, self.attention_head_size)

        self.is_decoder = config.is_decoder

    def transpose_for_scores(self, x: torch.Tensor) -> torch.Tensor:
        new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
        x = x.view(new_x_shape)
        return x.permute(0, 2, 1, 3)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor]:
        mixed_query_layer = self.query(hidden_states)

        # If this is instantiated as a cross-attention module, the keys
        # and values come from an encoder; the attention mask needs to be
        # such that the encoder's padding tokens are not attended to.
        is_cross_attention = encoder_hidden_states is not None

        if is_cross_attention and past_key_value is not None:
            # reuse k,v, cross_attentions
            key_layer = past_key_value[0]
            value_layer = past_key_value[1]
            attention_mask = encoder_attention_mask
        elif is_cross_attention:
            key_layer = self.transpose_for_scores(self.key(encoder_hidden_states))
            value_layer = self.transpose_for_scores(self.value(encoder_hidden_states))
            attention_mask = encoder_attention_mask
        elif past_key_value is not None:
            key_layer = self.transpose_for_scores(self.key(hidden_states))
            value_layer = self.transpose_for_scores(self.value(hidden_states))
            key_layer = torch.cat([past_key_value[0], key_layer], dim=2)
            value_layer = torch.cat([past_key_value[1], value_layer], dim=2)
        else:
            key_layer = self.transpose_for_scores(self.key(hidden_states))
            value_layer = self.transpose_for_scores(self.value(hidden_states))

        query_layer = self.transpose_for_scores(mixed_query_layer)

        use_cache = past_key_value is not None
        if self.is_decoder:
            # if cross_attention save Tuple(torch.Tensor, torch.Tensor) of all cross attention key/value_states.
            # Further calls to cross_attention layer can then reuse all cross-attention
            # key/value_states (first "if" case)
            # if uni-directional self-attention (decoder) save Tuple(torch.Tensor, torch.Tensor) of
            # all previous decoder key/value_states. Further calls to uni-directional self-attention
            # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
            # if encoder bi-directional self-attention `past_key_value` is always `None`
            past_key_value = (key_layer, value_layer)

        # Take the dot product between "query" and "key" to get the raw attention scores.
        attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))

        if self.position_embedding_type == "relative_key" or self.position_embedding_type == "relative_key_query":
            query_length, key_length = query_layer.shape[2], key_layer.shape[2]
            if use_cache:
                position_ids_l = torch.tensor(key_length - 1, dtype=torch.long, device=hidden_states.device).view(
                    -1, 1
                )
            else:
                position_ids_l = torch.arange(query_length, dtype=torch.long, device=hidden_states.device).view(-1, 1)
            position_ids_r = torch.arange(key_length, dtype=torch.long, device=hidden_states.device).view(1, -1)
            distance = position_ids_l - position_ids_r

            positional_embedding = self.distance_embedding(distance + self.max_position_embeddings - 1)
            positional_embedding = positional_embedding.to(dtype=query_layer.dtype)  # fp16 compatibility

            if self.position_embedding_type == "relative_key":
                relative_position_scores = torch.einsum("bhld,lrd->bhlr", query_layer, positional_embedding)
                attention_scores = attention_scores + relative_position_scores
            elif self.position_embedding_type == "relative_key_query":
                relative_position_scores_query = torch.einsum("bhld,lrd->bhlr", query_layer, positional_embedding)
                relative_position_scores_key = torch.einsum("bhrd,lrd->bhlr", key_layer, positional_embedding)
                attention_scores = attention_scores + relative_position_scores_query + relative_position_scores_key

        attention_scores = attention_scores / math.sqrt(self.attention_head_size)
        if attention_mask is not None:
            # Apply the attention mask is (precomputed for all layers in DummyBertModel forward() function)
            attention_scores = attention_scores + attention_mask

        # Normalize the attention scores to probabilities.
        attention_probs = nn.functional.softmax(attention_scores, dim=-1)

        # This is actually dropping out entire tokens to attend to, which might
        # seem a bit unusual, but is taken from the original Transformer paper.
        attention_probs = self.dropout(attention_probs)

        # Mask heads if we want to
        if head_mask is not None:
            attention_probs = attention_probs * head_mask

        context_layer = torch.matmul(attention_probs, value_layer)

        context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
        new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
        context_layer = context_layer.view(new_context_layer_shape)

        outputs = (context_layer, attention_probs) if output_attentions else (context_layer,)

        if self.is_decoder:
            outputs = outputs + (past_key_value,)


SUMMARY:
**Class Summary**

`DummyBertSelfAttention`
======================

A PyTorch implementation of the multi-head self-attention mechanism from the BERT model architecture.

**Primary Purpose**
-------------------

This class computes the attention scores and context layer for self-supervised learning or next sentence prediction tasks.

**Key Implementation Details**
-----------------------------

* Uses a multihead decoder attention mechanism with dimensionality reduction (dot product).
* Applies position embeddings for relative position encoding.
* Supports different types of position embeddings: `absolute`, `relative_key`, and `relative_key_query`.
* Masks heads using a provided head mask tensor, if applicable.

**Notable Dependencies or Requirements**
------------------------------------------

* `torch`: PyTorch library with neural network module (`nn`).
* `math.sqrt()`: Mathematical operations to compute square roots.
* `nn.functional.softmax()` : Softmax activation function for normalizing attention scores.
* `nn.Dropout()`: Dropout probability layer to prevent overfitting.

**Method Summary**
------------------

### `__init__(config, position_embedding_type=None)` init

 Initializes the module with configuration and optional position embedding type.

### `transpose_for_scores(x: torch.Tensor) -> torch.Tensor` transpose for scores

 Transposes input tensor for multihead self-attention mechanism computation.

### `forward(hidden_states: torch.Tensor, ...)` forward pass

 Computes attention scores and context layer for self-supervised learning or next sentence prediction tasks.
 
 Returns a tuple of `(context_layer, attention_probs)` if `output_attentions` is True; otherwise returns only the first element.


key: chroma:document
string_value:
CODE:

class DummyBertSdpaSelfAttention(DummyBertSelfAttention):
    def __init__(self, config, position_embedding_type=None):
        super().__init__(config, position_embedding_type=position_embedding_type)
        self.dropout_prob = config.attention_probs_dropout_prob
        self.require_contiguous_qkv = version.parse(get_torch_version()) < version.parse("2.2.0")

    # Adapted from DummyBertSelfAttention
    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor]:
        if self.position_embedding_type != "absolute" or output_attentions or head_mask is not None:
            # TODO: Improve this warning with e.g. `model.config._attn_implementation = "manual"` once implemented.
            logger.warning_once(
                "DummyBertSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support "
                "non-absolute `position_embedding_type` or `output_attentions=True` or `head_mask`. Falling back to "
                "the manual attention implementation, but specifying the manual implementation will be required from "
                "Transformers version v5.0.0 onwards. This warning can be removed using the argument "
                '`attn_implementation="eager"` when loading the model.'
            )
            return super().forward(
                hidden_states,
                attention_mask,
                head_mask,
                encoder_hidden_states,
                encoder_attention_mask,
                past_key_value,
                output_attentions,
            )

        bsz, tgt_len, _ = hidden_states.size()

        query_layer = self.transpose_for_scores(self.query(hidden_states))

        # If this is instantiated as a cross-attention module, the keys and values come from an encoder; the attention
        # mask needs to be such that the encoder's padding tokens are not attended to.
        is_cross_attention = encoder_hidden_states is not None

        current_states = encoder_hidden_states if is_cross_attention else hidden_states
        attention_mask = encoder_attention_mask if is_cross_attention else attention_mask

        # Check `seq_length` of `past_key_value` == `len(current_states)` to support prefix tuning
        if is_cross_attention and past_key_value and past_key_value[0].shape[2] == current_states.shape[1]:
            key_layer, value_layer = past_key_value
        else:
            key_layer = self.transpose_for_scores(self.key(current_states))
            value_layer = self.transpose_for_scores(self.value(current_states))
            if past_key_value is not None and not is_cross_attention:
                key_layer = torch.cat([past_key_value[0], key_layer], dim=2)
                value_layer = torch.cat([past_key_value[1], value_layer], dim=2)

        if self.is_decoder:
            # if cross_attention save Tuple(torch.Tensor, torch.Tensor) of all cross attention key/value_states.
            # Further calls to cross_attention layer can then reuse all cross-attention
            # key/value_states (first "if" case)
            # if uni-directional self-attention (decoder) save Tuple(torch.Tensor, torch.Tensor) of
            # all previous decoder key/value_states. Further calls to uni-directional self-attention
            # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
            # if encoder bi-directional self-attention `past_key_value` is always `None`
            past_key_value = (key_layer, value_layer)

        # SDPA with memory-efficient backend is broken in torch==2.1.2 when using non-contiguous inputs and a custom
        # attn_mask, so we need to call `.contiguous()` here. This was fixed in torch==2.2.0.
        # Reference: https://github.com/pytorch/pytorch/issues/112577
        if self.require_contiguous_qkv and query_layer.device.type == "cuda" and attention_mask is not None:
            query_layer = query_layer.contiguous()
            key_layer = key_layer.contiguous()
            value_layer = value_layer.contiguous()

        # We dispatch to SDPA's Flash Attention or Efficient kernels via this `is_causal` if statement instead of an inline conditional assignment
        # in SDPA to support both torch.compile's dynamic shapes and full graph options. An inline conditional prevents dynamic shapes from compiling.
        # The tgt_len > 1 is necessary to match with AttentionMaskConverter.to_causal_4d that does not create
        # a causal mask in case tgt_len == 1.
        is_causal = (
            True if self.is_decoder and not is_cross_attention and attention_mask is None and tgt_len > 1 else False
        )

        attn_output = torch.nn.functional.scaled_dot_product_attention(
            query_layer,
            key_layer,
            value_layer,
            attn_mask=attention_mask,
            dropout_p=self.dropout_prob if self.training else 0.0,
            is_causal=is_causal,
        )

        attn_output = attn_output.transpose(1, 2)
        attn_output = attn_output.reshape(bsz, tgt_len, self.all_head_size)

        outputs = (attn_output,)
        if self.is_decoder:
            outputs = outputs + (past_key_value,)


SUMMARY:
**Class Summary: DummyBertSdpaSelfAttention**

The `DummyBertSdpaSelfAttention` class extends the `DummyBertSelfAttention` class, inheriting its functionality. It is used to simulate the behavior of the transformers library's `scaled_dot_product_attention` function.

**Primary Purpose**

The primary purpose of this class is to handle non-absolute position embeddings and support torch.compile's dynamic shapes for fast inference.

**Key Implementation Details**

*   Uses `torch.nn.functional.scaled_dot_product_attention` to compute attention outputs.
*   Dispatches to SDPA's Flash Attention or Efficient kernels via an `is_causal` if statement.
*   Applies contouring operations for efficient computation on non-contiguous inputs.
*   Handles cross-attention and prefix tuning support.

**Notable Dependencies and Requirements**

*   Depends on the `DummyBertSelfAttention` class and uses its `forward` method as a base implementation.
*   Requires torch 2.2.0 or later due to fixes in SDPA's Flash Attention backend.

**(External References)**

-   `torch.nn.functional.scaled_dot_product_attention`
-   ` DummyBertSelfAttention` class
-   `config.attention_probs_dropout_prob`

Note: The exact reference to the corresponding Transformers library function mentioned above can be found at :https://github.com/huggingfacetransformers/Transformers


key: chroma:document
string_value:
CODE:

class DummyBertSelfOutput(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        hidden_states = self.LayerNorm(hidden_states + input_tensor)


SUMMARY:
**Summary**

This is a PyTorch implementation of a self-output transformation module for BERT, inspired by the Transformer architecture. The `DummyBertSelfOutput` class transforms the input-hidden-states tensor using a linear layer (`self.dense`), dropout regularization (`self.dropout`), and layer normalization (`self.LayerNorm`). The primary purpose is to normalize and conditionally scale the hidden states before outputting them.

**Key Implementation Details**

1. Linear transformation: `self.dense(nn.Linear(config.hidden_size, config.hidden_size))`
2. Dropout regularization: `self.dropout(config.hidden_dropout_prob)`
3. Layer Normalization: `self.LayerNorm(config.layer_norm_eps)`

**Notable Dependencies and Requirements**

* PyTorch (`torch`)
* Hugging Face Transformer library
* BERT configuration (`config`)


key: chroma:document
string_value:
CODE:

class DummyBertAttention(nn.Module):
    def __init__(self, config, position_embedding_type=None):
        super().__init__()
        self.self = DUMMY_BERT_SELF_ATTENTION_CLASSES[config._attn_implementation](
            config, position_embedding_type=position_embedding_type
        )
        self.output = DummyBertSelfOutput(config)
        self.pruned_heads = set()

    def prune_heads(self, heads):
        if len(heads) == 0:
            return
        heads, index = find_pruneable_heads_and_indices(
            heads, self.self.num_attention_heads, self.self.attention_head_size, self.pruned_heads
        )

        # Prune linear layers
        self.self.query = prune_linear_layer(self.self.query, index)
        self.self.key = prune_linear_layer(self.self.key, index)
        self.self.value = prune_linear_layer(self.self.value, index)
        self.output.dense = prune_linear_layer(self.output.dense, index, dim=1)

        # Update hyper params and store pruned heads
        self.self.num_attention_heads = self.self.num_attention_heads - len(heads)
        self.self.all_head_size = self.self.attention_head_size * self.self.num_attention_heads
        self.pruned_heads = self.pruned_heads.union(heads)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor]:
        self_outputs = self.self(
            hidden_states,
            attention_mask,
            head_mask,
            encoder_hidden_states,
            encoder_attention_mask,
            past_key_value,
            output_attentions,
        )
        attention_output = self.output(self_outputs[0], hidden_states)
        outputs = (attention_output,) + self_outputs[1:]  # add attentions if we output them


SUMMARY:
**Concise Summary:**

The `DummyBertAttention` class implements the BERT attention mechanism. It uses a `SelfAttention` module to compute attention weights and a `SelfOutput` module to apply the learned embedding.

**Key Implementation Details:**

*   **Attention Mechanism:** The `prune_heads` method applies attention head pruning based on the provided heads, updating hyperparameters accordingly.
*   **Forward Method:** The `forward` method first computes attention outputs using `self.self` and then passes them through `self.output`.
*   **Notable Dependencies:**

    *   `DummyBertSelfAttention_CLASSES`: A mapping class for BERT attention classes.
    *   `DummyBertSelfOutput`: A module for self-output operation.

**External Dependencies:**

*   `torch` library
*   `find_pruneable_heads_and_indices`, `prune_linear_layer`, and other utility functions in the codebase.


key: chroma:document
string_value:
CODE:

class DummyBertIntermediate(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
        if isinstance(config.hidden_act, str):
            self.intermediate_act_fn = ACT2FN[config.hidden_act]
        else:
            self.intermediate_act_fn = config.hidden_act

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.intermediate_act_fn(hidden_states)


SUMMARY:
**Summary**

The `DummyBertIntermediate` class inherits from PyTorch's `nn.Module`. It implements the intermediate layer of BERT, comprising:

* A linear dense layer (`self.dense`) with an input size of `config.hidden_size` and output size of `config.intermediate_size`.
* If `config.hidden_act` is a string, it calls the corresponding activation function from the `ACT2FN` dictionary. Otherwise, it uses `config.hidden_act` as the activation function (`self.intermediate_act_fn`).
* The forward pass applies self.dense to input `hidden_states`, followed by the application of `self.intermediate_act_fn`.

**Key Details**

* Input size: `config.hidden_size`
* Output size: `config(intermediate_size)`
* Activation function:
	+ If string, uses dictionary lookup
	+ Otherwise, uses `config.hidden_act`

**Dependencies & Requirements**

None explicitly mentioned. However, it's implied that `config` object contains the necessary configuration parameters (`hidden_size`, `intermediate_size`, and `hidden_act`). Additionally, PyTorch's neural network module (`nn.Module`) is required.


key: chroma:document
string_value:
CODE:

class DummyBertOutput(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        hidden_states = self.LayerNorm(hidden_states + input_tensor)


SUMMARY:
**Summary**

The `DummyBertOutput` class is an implementation of a PyTorch neural network module (`nn.Module`) designed to mimic the output layer of BERT (Bidirectional Encoder Representations from Transformers). It takes two inputs: `hidden_states` and `input_tensor`.

**Primary Purpose**
The main purpose of this class is to process `hidden_states` using a linear transformation, followed by a dropout regularization, and finally applying layer normalization with an additional input adjustment.

**Key Implementation Details**

*   Linear Transformation (`self.dense`)
    *   Input: `hidden_states` (torch.Tensor)
    *   Output: transformed tensor
*   Dropout Regularization (`self.dropout`)
    *   Input: transformed tensor from linear transformation
    *   Output: regularized tensor after dropout
*   Layer Normalization (`self.LayerNorm`)
    *   Input: combination of output from linear transformation, dropout regularization, and input_tensor
    *   Output: normalized tensor

**Notable Dependencies**

*   `nn.Module` (PyTorch neural network module)
*   `torch.Tensor`
*   `nn.Linear`, `nn.Dropout`, and `nn.LayerNorm` modules ( PyTorch linear and normalization layers)

**No external dependencies listed.**


key: chroma:document
string_value:
CODE:

class DummyBertLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.chunk_size_feed_forward = config.chunk_size_feed_forward
        self.seq_len_dim = 1
        self.attention = DummyBertAttention(config)
        self.is_decoder = config.is_decoder
        self.add_cross_attention = config.add_cross_attention
        if self.add_cross_attention:
            if not self.is_decoder:
                raise ValueError(f"{self} should be used as a decoder model if cross attention is added")
            self.crossattention = DummyBertAttention(config, position_embedding_type="absolute")
        self.intermediate = DummyBertIntermediate(config)
        self.output = DummyBertOutput(config)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor]:
        # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
        self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
        self_attention_outputs = self.attention(
            hidden_states,
            attention_mask,
            head_mask,
            output_attentions=output_attentions,
            past_key_value=self_attn_past_key_value,
        )
        attention_output = self_attention_outputs[0]

        # if decoder, the last output is tuple of self-attn cache
        if self.is_decoder:
            outputs = self_attention_outputs[1:-1]
            present_key_value = self_attention_outputs[-1]
        else:
            outputs = self_attention_outputs[1:]  # add self attentions if we output attention weights

        cross_attn_present_key_value = None
        if self.is_decoder and encoder_hidden_states is not None:
            if not hasattr(self, "crossattention"):
                raise ValueError(
                    f"If `encoder_hidden_states` are passed, {self} has to be instantiated with cross-attention layers"
                    " by setting `config.add_cross_attention=True`"
                )

            # cross_attn cached key/values tuple is at positions 3,4 of past_key_value tuple
            cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
            cross_attention_outputs = self.crossattention(
                attention_output,
                attention_mask,
                head_mask,
                encoder_hidden_states,
                encoder_attention_mask,
                cross_attn_past_key_value,
                output_attentions,
            )
            attention_output = cross_attention_outputs[0]
            outputs = outputs + cross_attention_outputs[1:-1]  # add cross attentions if we output attention weights

            # add cross-attn cache to positions 3,4 of present_key_value tuple
            cross_attn_present_key_value = cross_attention_outputs[-1]
            present_key_value = present_key_value + cross_attn_present_key_value

        layer_output = apply_chunking_to_forward(
            self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output
        )
        outputs = (layer_output,) + outputs

        # if decoder, return the attn key/values as the last output
        if self.is_decoder:
            outputs = outputs + (present_key_value,)

        return outputs

    def feed_forward_chunk(self, attention_output):
        intermediate_output = self.intermediate(attention_output)
        layer_output = self.output(intermediate_output, attention_output)


SUMMARY:
**Class Summary: DummyBertLayer**

The `DummyBertLayer` class represents a single layer in the BERT architecture. It is a PyTorch module that utilizes multi-head self-attention and feed-forward networks.

**Primary Purpose:** To process input data through self-attention and feed-forward networks, producing output features.

**Key Implementation Details:**

*   The `forward` method processes input tensors for self-attention and cross-attention. It returns a tuple of output features.
*   Two critical components are the `DummyBertAttention` and `DummyBertIntermediate` modules, which implement attention mechanisms and feed-forward networks, respectively.

**Notable Dependencies or Requirements:**

*   Uses `torch.Tensor`, `Optional[ torch.FloatTensor]`, and other PyTorch-related classes/types for tensor operations.
*   Incorporates dependency on `DummyBertAttention` and `DummyBertIntermediate` modules (not shown in this code snippet).


key: chroma:document
string_value:
CODE:

class DummyBertEncoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.layer = nn.ModuleList([DummyBertLayer(config) for _ in range(config.num_hidden_layers)])
        self.gradient_checkpointing = False

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = False,
        output_hidden_states: Optional[bool] = False,
        return_dict: Optional[bool] = True,
    ) -> Union[Tuple[torch.Tensor], BaseModelOutputWithPastAndCrossAttentions]:
        all_hidden_states = () if output_hidden_states else None
        all_self_attentions = () if output_attentions else None
        all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None

        if self.gradient_checkpointing and self.training:
            if use_cache:
                logger.warning_once(
                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
                )
                use_cache = False

        next_decoder_cache = () if use_cache else None
        for i, layer_module in enumerate(self.layer):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            layer_head_mask = head_mask[i] if head_mask is not None else None
            past_key_value = past_key_values[i] if past_key_values is not None else None

            if self.gradient_checkpointing and self.training:
                layer_outputs = self._gradient_checkpointing_func(
                    layer_module.__call__,
                    hidden_states,
                    attention_mask,
                    layer_head_mask,
                    encoder_hidden_states,
                    encoder_attention_mask,
                    past_key_value,
                    output_attentions,
                )
            else:
                layer_outputs = layer_module(
                    hidden_states,
                    attention_mask,
                    layer_head_mask,
                    encoder_hidden_states,
                    encoder_attention_mask,
                    past_key_value,
                    output_attentions,
                )

            hidden_states = layer_outputs[0]
            if use_cache:
                next_decoder_cache += (layer_outputs[-1],)
            if output_attentions:
                all_self_attentions = all_self_attentions + (layer_outputs[1],)
                if self.config.add_cross_attention:
                    all_cross_attentions = all_cross_attentions + (layer_outputs[2],)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(
                v
                for v in [
                    hidden_states,
                    next_decoder_cache,
                    all_hidden_states,
                    all_self_attentions,
                    all_cross_attentions,
                ]
                if v is not None
            )
        return BaseModelOutputWithPastAndCrossAttentions(
            last_hidden_state=hidden_states,
            past_key_values=next_decoder_cache,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
            cross_attentions=all_cross_attentions,


SUMMARY:
**Concise Summary:**

The `DummyBertEncoder` class is an encoder layer for the BERT model, implementing the `nn.Module` interface. It takes in various input tensors (`hidden_states`, `attention_mask`, etc.) and transforms them through self-attention mechanisms.

**Key Implementation Details:**

* The encoder consists of multiple identical `DummyBertLayer` instances wrapped within a `ModuleList`.
* Each layer runs through a forward pass, applying self-attention to the input `hidden_states`. If gradient checkpointing is enabled, it falls back to a fallback implementation.
* After each layer's forward pass, it updates the cache and attentions outputs.

**Notable Dependencies and Requirements:**

* The class depends on `nn.Module`, `torch.Tensor`, and various PyTorch modules (e.g., `nn.ModuleList`, `Optional`).
* It uses `logger.warning_once` for logging warnings related to gradient checkpointing.
* The class implements the `BaseModelOutputWithPastAndCrossAttentions` interface, which suggests an integration with larger PyTorch-based models.
* External dependencies are implicitly required through the use of PyTorch modules and classes.


key: chroma:document
string_value:
CODE:

class DummyBertPooler(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.activation = nn.Tanh()

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.
        first_token_tensor = hidden_states[:, 0]
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)


SUMMARY:
**Summary:**

`DummyBertPooler` is a PyTorch neural network module implementing a simplified BERT pooler. It takes in `hidden_states` and outputs the pooled representation of the first token using a linear transformation (`self.dense`) followed by the Tanh activation function (`self.activation`). 

**Key Details:**
- Takes `config.hidden_size` as input and output.
- Utilizes `nn.Linear` for linear transformation and `nn.Tanh` for activation function.

**Dependencies:** 
- References `torch.Tensor`
- Inherited interfaces: `nn.Module`, `nn.Linear`

**Notable Requirements:**
- Inherits from PyTorch's neural network module (`nn.Module`)
- Import of specific activation functions (`nn.Tanh`) and linear transformation (`nn.Linear`).


key: chroma:document
string_value:
CODE:

def load_tf_weights_in_dummy_bert(model, config, tf_checkpoint_path):
    """Load tf checkpoints in a pytorch model."""
    try:
        import re

        import numpy as np
        import tensorflow as tf
    except ImportError:
        logger.error(
            "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see "
            "https://www.tensorflow.org/install/ for installation instructions."
        )
        raise
    tf_path = os.path.abspath(tf_checkpoint_path)
    logger.info(f"Converting TensorFlow checkpoint from {tf_path}")
    # Load weights from TF model
    init_vars = tf.train.list_variables(tf_path)
    names = []
    arrays = []
    for name, shape in init_vars:
        logger.info(f"Loading TF weight {name} with shape {shape}")
        array = tf.train.load_variable(tf_path, name)
        names.append(name)
        arrays.append(array)

    for name, array in zip(names, arrays):
        name = name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(
            n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
            for n in name
        ):
            logger.info(f"Skipping {'/'.join(name)}")
            continue
        pointer = model
        for m_name in name:
            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
                scope_names = re.split(r"_(\d+)", m_name)
            else:
                scope_names = [m_name]
            if scope_names[0] == "kernel" or scope_names[0] == "gamma":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
                pointer = getattr(pointer, "bias")
            elif scope_names[0] == "output_weights":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "squad":
                pointer = getattr(pointer, "classifier")
            else:
                try:
                    pointer = getattr(pointer, scope_names[0])
                except AttributeError:
                    logger.info(f"Skipping {'/'.join(name)}")
                    continue
            if len(scope_names) >= 2:
                num = int(scope_names[1])
                pointer = pointer[num]
        if m_name[-11:] == "_embeddings":
            pointer = getattr(pointer, "weight")
        elif m_name == "kernel":
            array = np.transpose(array)
        try:
            if pointer.shape != array.shape:
                raise ValueError(f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched")
        except ValueError as e:
            e.args += (pointer.shape, array.shape)
            raise
        logger.info(f"Initialize PyTorch weight {name}")
        pointer.data = torch.from_numpy(array)


SUMMARY:
**Summary: Loading TensorFlow Weights in PyTorch Model**

```python
def load_tf_weights_in_dummy_bert(model, config, tf_checkpoint_path):
    """
    Load pre-trained TensorFlow weights into a PyTorch model.

    Args:
    - model (PyTorch model): The model to load weights into.
    - config: (not used)
    - tf_checkpoint_path: Path to the TensorFlow checkpoint file.

    Returns:
    - The loaded PyTorch model with pre-trained weights.
    """

    **Implementation Details**

    1. Loads TensorFlow checkpoints using `tf.train.list_variables` and `tf.train.load_variable`.
    2. Skips certain variables (e.g., `adam_v`, `adam_m`) that are not required for using the pre-trained model.
    3. Uses NumPy to transpose array `array` when its name ends with `_embeddings`.
    4. Checks shape mismatch between `pointer` and `array` using `ValueError`.

**Design Choice:**

* The function uses a pointer-based approach to navigate the PyTorch model's namespace.
* This design choice enables efficient loading of TensorFlow weights into the PyTorch model.

**Dependencies:**

* TensorFlow (`tf`)
* NumPy (`np`)
* PyTorch (`torch`)

**Interfaces:**

* `model`: The PyTorch model to load weights into (instance-specific).
* `logger`: A logging interface for reporting progress and errors (not shown in the code snippet).


key: chroma:document
string_value:
CODE:

class DummyBertPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """

    config_class = DummyBertConfig
    load_tf_weights = load_tf_weights_in_dummy_bert
    base_model_prefix = "dummy_bert"
    supports_gradient_checkpointing = True
    _supports_sdpa = True

    def _init_weights(self, module):
        """Initialize the weights"""
        if isinstance(module, nn.Linear):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
            if module.bias is not None:
                module.bias.data.zero_()
        elif isinstance(module, nn.Embedding):
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
            if module.padding_idx is not None:
                module.weight.data[module.padding_idx].zero_()
        elif isinstance(module, nn.LayerNorm):
            module.bias.data.zero_()


SUMMARY:
**Summary:**

`DummyBertPreTrainedModel`: An abstract class implementing a simplified interface for downloading and loading pre-trained models.

**Main Purpose and Functionality:**
This class provides a basic framework for handling weights initialization, allowing users to download and load pre-trained models.

**Key Implementation Details:**

* Initializes model weights using a normal distribution with mean 0.0 and standard deviation `config.initializer_range`.
* Handles initialization for linear layers, embedding tables, and layer normalization modules.
* Copies behavior from the PyTorch `load_tf_weights_in_dummy_bert` function.

**Notable Dependencies or Requirements:**

* `DummyBertConfig`: Config class for this model (not shown in this snippet).
* `nn.Linear`, `nn.Embedding`, `nn.LayerNorm`: PyTorch neural network module classes.
* `load_tf_weights_in_dummy_bert`: Function imported from an external library, used for loading TensorFlow weights.

**Interface Inheritance:**
Inherits from `PreTrainedModel`, providing a simplified interface for downloading and loading pre-trained models.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/configuration_new_model.py


key: chroma:document
string_value:
CODE:

class NewModelConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`NewModelModel`]. It is used to instantiate an NewModel
    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
    defaults will yield a similar configuration to that of the NewModel-7B.
    e.g. [google/new_model-7b](https://huggingface.co/google/new_model-7b)
    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.
    Args:
        vocab_size (`int`, *optional*, defaults to 256000):
            Vocabulary size of the NewModel model. Defines the number of different tokens that can be represented by the
            `inputs_ids` passed when calling [`NewModelModel`]
        hidden_size (`int`, *optional*, defaults to 3072):
            Dimension of the hidden representations.
        intermediate_size (`int`, *optional*, defaults to 24576):
            Dimension of the MLP representations.
        num_hidden_layers (`int`, *optional*, defaults to 28):
            Number of hidden layers in the Transformer decoder.
        num_attention_heads (`int`, *optional*, defaults to 16):
            Number of attention heads for each attention layer in the Transformer decoder.
        num_key_value_heads (`int`, *optional*, defaults to 16):
            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
            by meanpooling all the original heads within that group. For more details checkout [this
            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
            `num_attention_heads`.
        head_dim (`int`, *optional*, defaults to 256):
            The attention head dimension.
        hidden_act (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`):
            The legacy activation function. It is overwritten by the `hidden_activation`.
        hidden_activation (`str` or `function`, *optional*):
            The non-linear activation function (function or string) in the decoder. Will default to `"gelu_pytorch_tanh"`
            if not specified. `"gelu_pytorch_tanh"` uses an approximation of the `"gelu"` activation function.
        max_position_embeddings (`int`, *optional*, defaults to 8192):
            The maximum sequence length that this model might ever be used with.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
            The epsilon used by the rms normalization layers.
        use_cache (`bool`, *optional*, defaults to `True`):
            Whether or not the model should return the last key/values attentions (not used by all models). Only
            relevant if `config.is_decoder=True`.
        pad_token_id (`int`, *optional*, defaults to 0):
            Padding token id.
        eos_token_id (`int`, *optional*, defaults to 1):
            End of stream token id.
        bos_token_id (`int`, *optional*, defaults to 2):
            Beginning of stream token id.
        tie_word_embeddings (`bool`, *optional*, defaults to `True`):
            Whether to tie weight embeddings
        rope_theta (`float`, *optional*, defaults to 10000.0):
            The base period of the RoPE embeddings.
        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
            Whether to use a bias in the query, key, value and output projection layers during self-attention.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities.
    ```python
    >>> from transformers import NewModelModel, NewModelConfig
    >>> # Initializing a NewModel new_model-7b style configuration
    >>> configuration = NewModelConfig()
    >>> # Initializing a model from the new_model-7b style configuration
    >>> model = NewModelModel(configuration)
    >>> # Accessing the model configuration
    >>> configuration = model.config
    ```"""

    model_type = "new_model"
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(
        self,
        vocab_size=256030,
        hidden_size=64,
        intermediate_size=90,
        num_hidden_layers=28,
        num_attention_heads=16,
        num_key_value_heads=16,
        head_dim=256,
        hidden_act="gelu_pytorch_tanh",
        hidden_activation=None,
        max_position_embeddings=1500,
        initializer_range=0.02,
        rms_norm_eps=1e-6,
        use_cache=True,
        pad_token_id=0,
        eos_token_id=1,
        bos_token_id=2,
        tie_word_embeddings=True,
        rope_theta=10000.0,
        attention_bias=False,
        attention_dropout=0.0,
        **kwargs,
    ):
        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.head_dim = head_dim
        self.num_key_value_heads = num_key_value_heads
        self.hidden_act = hidden_act
        self.hidden_activation = hidden_activation
        self.initializer_range = initializer_range
        self.rms_norm_eps = rms_norm_eps
        self.use_cache = use_cache
        self.rope_theta = rope_theta
        self.attention_bias = attention_bias
        self.attention_dropout = attention_dropout

    @property
    def num_heads(self):


SUMMARY:
Here is a concise and focused summary of the class code chunk:

**Class:** `NewModelConfig`
**Purpose:** The configuration class for storing parameters defining a [`NewModelModel`].

**Main Functionality:**
This class enables instantiation of a `NewModel` model according to specified arguments, including vocabulary size, hidden dimensions, number of layers, attention heads, and other hyperparameters.

**Key Implementation Details:**

* **Attributes:** Vocabulary size (`vocab_size`), hidden dimension sizes (`hidden_size`, `intermediate_size`), number of layers (`num_hidden_layers`), attention head count (`num_attention_heads`), head dimension (`head_dim`), activation functions (`hidden_act`, `hidden_activation`), and other hyperparameters.
* **Constructor:** Initializes the class with an optional dictionary of hyperparameters.

**Notable Dependencies and Requirements:**

* Inherits from [`PretrainedConfig`]
* Uses external dependencies for weight initialization (e.g., truncated normal initializer)
* Implements Grouped Query Attention and RoPE embeddings using hyperparameter settings (`num_key_value_heads`, `rope_theta`)


key: chroma:document
string_value:
CODE:
# example where we need some deps and some functions
class SuperModel(LlamaModel):
    def forward(
        self,
        input_ids: torch.LongTensor = None,
        attention_mask: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None,
        inputs_embeds: Optional[torch.FloatTensor] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        cache_position: Optional[torch.LongTensor] = None,
    ) -> Union[Tuple, CausalLMOutputWithPast]:
        out = super().forward(
            input_ids,
            attention_mask,
            position_ids,
            past_key_values,
            inputs_embeds,
            use_cache,
            output_attentions,
            output_hidden_states,
            return_dict,
            cache_position,
        )
        out.logits *= 2**4


SUMMARY:
**Class Summary: `SuperModel`**

### Main Purpose and Functionality

The `SuperModel` class extends `LlamaModel` and inherits its functionality. It provides an improved forward pass that includes additional computations.

### Key Implementation Details

* **Forward pass**: The model's input is passed through a modified call to the parent class's `forward` method.
* **Linear transformation**: The output logits are scaled by a factor of 2^4 (16).

### Notable Dependencies and Requirements

The following dependencies and interfaces are used in this code:

* `torch`: PyTorch library for tensor operations
* `LlamaModel`: Parent class with its own `forward` method implementation
* `Cache`, `List[torch.FloatTensor]`: Optional types in the PyTorch library
* `CausalLMOutputWithPast`: Expected return type of the `forward` method

### Interface and Inheritance

The `SuperModel` class inherits from `LlamaModel`, suggesting that it is designed to build upon or extend the parent class's functionality.

### External Dependencies (none mentioned in this code snippet)


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_new_task_model.py


key: chroma:document
string_value:
CODE:

class NewTaskModelMultiModalProjector(nn.Module):
    def __init__(self, config: NewTaskModelConfig):
        super().__init__()
        self.linear = nn.Linear(config.vision_config.hidden_size, config.vision_config.projection_dim, bias=True)

    def forward(self, image_features):
        hidden_states = self.linear(image_features)



SUMMARY:
**Summary**

`NewTaskModelMultiModalProjector`: A multi-modal projector class extending PyTorch's `nn.Module`. This model projects input features from various modalities onto a shared high-dimensional space.

**Key Details:**

* **Forward Pass**: Takes in image features, applies a linear transformation, and returns the projected output.
	+ Input: `image_features`
	+ `linear` = `nn.Linear(config.vision_config.hidden_size, config.vision_config.projection_dim, bias=True)`
* **Configured Linear Layer**: A fully connected layer with:
	+ Input dimension equal to `config.vision_config.hidden_size`
	+ Output dimension equal to `config.vision_config.projection_dim`
	+ Includes bias term
* **No External Dependencies**

This summary highlights the primary purpose and key implementation details of the class, including its forward pass, configured linear layer, and lack of external dependencies.


key: chroma:document
string_value:
CODE:

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 4]
    x2 = x[..., x.shape[-1] // 4 :]


SUMMARY:
**Summary:**

Rotates half of the hidden dimensions in the input tensor `x` by shifting and concatenating two adjacent segments.

**Key Details:**

* Rotates only half of the hidden dimensions (top 4 dimensions).
* Shifts top part to bottom and vice versa using concatenation:
	+ `x1 = x[..., : x.shape[-1] // 4]`
	+ `-x2 = -x[..., x.shape[-1] // 4 :]`
* Uses PyTorch tensor manipulation, specifically `torch.cat`.

**Notable Dependencies:**

* Requires PyTorch library (`import torch`)
* Assumes input tensor `x` has shape that supports splitting along the last dimension.


key: chroma:document
string_value:
CODE:
# example where we need some deps and some functions
class DummyModel(LlamaModel):


SUMMARY:
**Summary**

This Java code defines a `DummyModel` class that extends the `LlamaModel` class. The main purpose of this class is to serve as a placeholder or example model.

**Key Implementation Details:**

* Extends `LlamaModel`, indicating inheritance of methods and properties.
* An empty class (`pass`), suggesting no specific implementation or functionality added on top of the base class.

**Notable Dependencies/Requirements:**

* References the `LlamaModel` class, implying that it relies on some functionality provided by this parent class.

**Additional Information:**
This code may be part of a larger model or library system where custom models are defined to interact with LlamaModel.


key: chroma:document
string_value:
CODE:

class RobertaEmbeddings(BertEmbeddings):
    def __init__(self, config):
        super().__init__(config)
        self.pad_token_id = config.pad_token_id
        self.position_embeddings = nn.Embedding(
            config.max_position_embeddings, config.hidden_size, config.pad_token_id


SUMMARY:
**Class Summary: RobertaEmbeddings**

The `RobertaEmbeddings` class extends the `BertEmbeddings` base class and is used for obtaining embeddings from input sequences in a Hugging Face's transformers library.

**Primary Purpose:** To compute sequence embeddings using a custom embedding layer, inheriting properties from the parent `BertEmbeddings` class.

**Key Implementation Details:**

* Initializes the embedding layer with a specific configuration (`config.max_position_embeddings`, `config.hidden_size`, and `config.pad_token_id`)
* Utilizes an `nn.Embedding` module to store position embeddings

**Notable Dependencies/Requirements:**

* Inherits from `BertEmbeddings`, utilizing its properties
* Depends on the Hugging Face's transformers library for the embedding layer (`nn.Embedding`)


key: chroma:document
string_value:
CODE:

class RobertaModel(BertModel):
    def __init__(self, config, add_pooling_layer=True):


SUMMARY:
**Summary: `RobertaModel` Class**

### Purpose and Functionality:

The `RobertaModel` class is an extension of the `BertModel`, inheriting its functionality. It is designed to accommodate the Roberta language model architecture, allowing for efficient generation and prediction tasks.

### Key Implementation Details:

- Initializes with a `config` dictionary and optionally adds a pooling layer (`add_pooling_layer=True`)
- Calls the parent class's (`BertModel`) initializer with the inherited parameters

### Dependencies and Requirements:

- Inherited from `BertModel`
- Extends Roberta language model architecture


key: chroma:document
string_value:
CODE:

class DummyBertModel(BertModel):
    def forward(
        self,
        input_ids: Optional[torch.Tensor] = None,
        attention_mask: Optional[torch.Tensor] = None,
        token_type_ids: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.Tensor] = None,
        head_mask: Optional[torch.Tensor] = None,
        inputs_embeds: Optional[torch.Tensor] = None,
        encoder_hidden_states: Optional[torch.Tensor] = None,
        encoder_attention_mask: Optional[torch.Tensor] = None,
        past_key_values: Optional[List[torch.FloatTensor]] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[torch.Tensor], BaseModelOutputWithPoolingAndCrossAttentions]:


SUMMARY:
**Class Summary: DummyBertModel**

The `DummyBertModel` class is a subclass of `BertModel` that overrides the `forward` method to simplify its behavior.

**Primary Purpose:**
The model's primary purpose is to forward input tensors through a basic BERT-like processing pipeline, effectively reducing its functionality by always returning `super().forward(input_ids)`.

**Key Implementation Details:**

*   **Method Signature:** The `forward` method accepts a variety of optional input tensors and returns either a tensor or a `BaseModelOutputWithPoolingAndCrossAttentions` object.
*   **Inherited Behavior:** The model calls the parent class's `forward` method directly with only the `input_ids` argument, implying that all other arguments are ignored.

**Notable Dependencies:**

*   `BertModel`
*   `torch`
        *   External Dependency: `torch`

**Interfaces:**

*   Inherited from `BertModel`: `BaseModelOutputWithPoolingAndCrossAttentions`.


key: chroma:document
string_value:
CODE:

class Multimodal2VisionAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(self, config):
        super().__init__()
        self.config = config
        self.embed_dim = config.hidden_size
        self.num_heads = config.num_attention_heads
        self.head_dim = self.embed_dim // self.num_heads
        if self.head_dim * self.num_heads != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`:"
                f" {self.num_heads})."
            )
        self.scale = self.head_dim**-0.5
        self.dropout = config.attention_dropout

        self.k_proj = nn.Linear(self.embed_dim, self.embed_dim)
        self.v_proj = nn.Linear(self.embed_dim, self.embed_dim)
        self.q_proj = nn.Linear(self.embed_dim, self.embed_dim)
        self.out_proj = nn.Linear(self.embed_dim, self.embed_dim)

    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        causal_attention_mask: Optional[torch.Tensor] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        """Input shape: Batch x Time x Channel"""

        bsz, tgt_len, embed_dim = hidden_states.size()

        # get query proj
        query_states = self.q_proj(hidden_states) * self.scale
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

        proj_shape = (bsz * self.num_heads, -1, self.head_dim)
        query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
        key_states = key_states.view(*proj_shape)
        value_states = value_states.view(*proj_shape)

        src_len = key_states.size(1)
        attn_weights = torch.bmm(query_states, key_states.transpose(1, 2))

        if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
            raise ValueError(
                f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
                f" {attn_weights.size()}"
            )

        # apply the causal_attention_mask first
        if causal_attention_mask is not None:
            if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
                raise ValueError(
                    f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is"
                    f" {causal_attention_mask.size()}"
                )
            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + causal_attention_mask
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        if attention_mask is not None:
            if attention_mask.size() != (bsz, 1, tgt_len, src_len):
                raise ValueError(
                    f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}"
                )
            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        attn_weights = nn.functional.softmax(attn_weights, dim=-1)

        if output_attentions:
            # this operation is a bit akward, but it's required to
            # make sure that attn_weights keeps its gradient.
            # In order to do so, attn_weights have to reshaped
            # twice and have to be reused in the following
            attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
        else:
            attn_weights_reshaped = None

        attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training)

        attn_output = torch.bmm(attn_probs, value_states)

        if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
            raise ValueError(
                f"`attn_output` should be of size {(bsz, self.num_heads, tgt_len, self.head_dim)}, but is"
                f" {attn_output.size()}"
            )

        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
        attn_output = attn_output.transpose(1, 2)
        attn_output = attn_output.reshape(bsz, tgt_len, embed_dim)

        attn_output = self.out_proj(attn_output)



SUMMARY:
**Class Summary: Multimodal2VisionAttention**
=============================================

*   **Main Purpose and Functionality:** This class implements the Multi-Head Attention mechanism as described in the "Attention Is All You Need" paper. It takes in input `hidden_states` and optional masks (`attention_mask` and `causal_attention_mask`) to compute attention weights and outputs.
*   **Key Implementation Details:**
    *   Uses a linear transformation followed by a scaling factor to project queries, keys, and values.
    *   Computes attention weights using a dot product and softmax function.
    *   Applies dropout to the attention weights.
    *   Transposes and reshapes the output for broadcasting.
*   **Notable Dependencies or Requirements:**
    *   `config`: Configuration object containing hidden size, number of heads, and attention dropouts.
    *   `nn.Module`: Base PyTorch module class.
    *   `torch.Tensor`: PyTorch tensor class.
    *   `nn.functional`: PyTorch neural network functional module.

**Method Signature:**
```python
forward(hidden_states: torch.Tensor, 
         attention_mask: Optional[torch.Tensor] = None, 
         causal_attention_mask: Optional[torch.Tensor] = None, 
         output_attentions: Optional[bool] = False) -> Tuple[torch.Tensor, Optional[torch.Tensor]]
```

**Example Use Case:**
```python
multimodal2vision_attention = Multimodal2VisionAttention(config)
hidden_states = ...
attention_mask = ...

attn_output, attn_weights = multimodal2vision_attention(hidden_states, attention_mask, None, False)
```


key: chroma:document
string_value:
CODE:

class Multimodal2VisionSdpaAttention(Multimodal2VisionAttention):
    """
    SDPA attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from
    `Multimodal2VisionAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to
    SDPA API.
    """

    # Adapted from Multimodal2VisionAttention.forward
    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        causal_attention_mask: Optional[torch.Tensor] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        if output_attentions:
            # TODO: Improve this warning with e.g. `model.config.attn_implementation = "manual"` once this is implemented.
            logger.warning_once(
                "Multimodal2VisionModel is using Multimodal2VisionSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not "
                "support `output_attentions=True`. Falling back to the manual attention implementation, but specifying "
                "the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can "
                'be removed using the argument `attn_implementation="eager"` when loading the model.'
            )
            return super().forward(
                hidden_states=hidden_states,
                attention_mask=attention_mask,
                causal_attention_mask=causal_attention_mask,
                output_attentions=output_attentions,
            )

        # MULTIMODAL2_VISION text model uses both `causal_attention_mask` and `attention_mask`
        if attention_mask is not None and causal_attention_mask is not None:
            attn_mask = attention_mask + causal_attention_mask
        elif causal_attention_mask is not None:
            attn_mask = causal_attention_mask
        else:
            attn_mask = attention_mask

        bsz, tgt_len, embed_dim = hidden_states.size()

        query_states = self.q_proj(hidden_states)
        key_states = self.k_proj(hidden_states)
        value_states = self.v_proj(hidden_states)

        query_states = query_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)
        key_states = key_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)
        value_states = value_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)

        # SDPA with memory-efficient backend is currently (torch==2.1.2) bugged with non-contiguous inputs with custom attn_mask,
        # Reference: https://github.com/pytorch/pytorch/issues/112577.
        if not is_torch_greater_or_equal_than_2_2 and query_states.device.type == "cuda" and attn_mask is not None:
            query_states = query_states.contiguous()
            key_states = key_states.contiguous()
            value_states = value_states.contiguous()

        # MULTIMODAL2_VISION text model uses both `causal_attention_mask` and `attention_mask` sequentially.
        attn_output = torch.nn.functional.scaled_dot_product_attention(
            query_states,
            key_states,
            value_states,
            attn_mask=attn_mask,
            dropout_p=self.dropout if self.training else 0.0,
            scale=self.scale,
        )

        attn_output = attn_output.transpose(1, 2)
        attn_output = attn_output.reshape(bsz, tgt_len, embed_dim)

        attn_output = self.out_proj(attn_output)



SUMMARY:
**Multimodal2VisionSdpaAttention Class Summary**

**Main Purpose and Functionality:**
The `Multimodal2VisionSdpaAttention` class is an attention module derived from `Multimodal2VisionAttention`, utilizing the `torch.nn.functional.scaled_dot_product_attention` function. It adapts the forward pass to work with the SDPA API.

**Key Implementation Details:**

*   Implements scaled dot-product attention using `torch.nn.functional.scaled_dot_product_attention`.
*   Handles both `causal_attention_mask` and `attention_mask` in the input.
*   Performs dimension-wise transposition of query, key, and value states before passing them to the scaled dot-product attention function.
*   Applies an optional scaling factor (`self.scale`) to the attention output.

**Notable Dependencies or Requirements:**

*   `torch.nn.functional.scaled_dot_product_attention`
*   `Multimodal2VisionAttention` class (with unchanged weights)
*   `torch` version 2.1.2+ for memory-efficient backend support
*   `self.dropout`, `self.scale`, and other attributes from `Multimodal2VisionAttention`

**Interface Summary:**

The class has the following interface:

*   `forward(hidden_states, attention_mask, causal_attention_mask, output_attentions) -> (attn_output, None)`
    *   Returns the scaled dot-product attention output.
    *   If `output_attentions` is set to `True`, falls back to a manual attention implementation.


key: chroma:document
string_value:
CODE:

class Multimodal2VisionFlashAttention2(Multimodal2VisionAttention):
    """
    Multimodal2VisionAttention flash attention module. This module inherits from `Multimodal2VisionAttention` as the weights of the module stays
    untouched. The only required change would be on the forward pass where it needs to correctly call the public API of
    flash attention and deal with padding tokens in case the input contains any of them.
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        # TODO: Should be removed once Flash Attention for RoCm is bumped to 2.1.
        # flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. Reference: https://github.com/Dao-AILab/flash-attention/releases/tag/v2.1.0.
        # Beware that with flash_attn<2.1, using q_seqlen != k_seqlen (except for the case q_seqlen == 1) produces a wrong mask (top-left).
        self._flash_attn_uses_top_left_mask = not is_flash_attn_greater_or_equal_2_10()

    # Adapted from transformers.models.llama.modeling_llama.LlamaFlashAttention2.forward
    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        causal_attention_mask: Optional[torch.Tensor] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        output_attentions = False

        batch_size, q_len, _ = hidden_states.size()

        query_states = self.q_proj(hidden_states)
        key_states = self.k_proj(hidden_states)
        value_states = self.v_proj(hidden_states)

        # Flash attention requires the input to have the shape
        # batch_size x seq_length x head_dim x hidden_dim
        # therefore we just need to keep the original shape
        query_states = query_states.view(batch_size, q_len, self.num_heads, self.head_dim)
        key_states = key_states.view(batch_size, q_len, self.num_heads, self.head_dim)
        value_states = value_states.view(batch_size, q_len, self.num_heads, self.head_dim)

        dropout_rate = self.dropout if self.training else 0.0

        # In PEFT, usually we cast the layer norms in float32 for training stability reasons
        # therefore the input hidden states gets silently casted in float32. Hence, we need
        # cast them back in the correct dtype just to be sure everything works as expected.
        # This might slowdown training & inference so it is recommended to not cast the LayerNorms
        # in fp32.

        input_dtype = query_states.dtype
        if input_dtype == torch.float32:
            if torch.is_autocast_enabled():
                target_dtype = torch.get_autocast_gpu_dtype()
            # Handle the case where the model is quantized
            elif hasattr(self.config, "_pre_quantization_dtype"):
                target_dtype = self.config._pre_quantization_dtype
            else:
                target_dtype = self.q_proj.weight.dtype

            logger.warning_once(
                f"The input hidden states seems to be silently casted in float32, this might be related to"
                f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in"
                f" {target_dtype}."
            )

            query_states = query_states.to(target_dtype)
            key_states = key_states.to(target_dtype)
            value_states = value_states.to(target_dtype)

        attn_output = _flash_attention_forward(
            query_states,
            key_states,
            value_states,
            attention_mask,
            q_len,
            dropout=dropout_rate,
            is_causal=causal_attention_mask is not None,
            use_top_left_mask=self._flash_attn_uses_top_left_mask,
        )

        attn_output = attn_output.reshape(batch_size, q_len, self.embed_dim).contiguous()
        attn_output = self.out_proj(attn_output)

        if not output_attentions:
            attn_weights = None



SUMMARY:
**Class Summary: Multimodal2VisionFlashAttention2**

The `Multimodal2VisionFlashAttention2` class is an extension of the `Multimodal2VisionAttention` class, integrating Flash Attention. It maintains the same weights but adapts to Flash Attention's requirements.

**Key Implementations:**

1. **Forward Pass**: The forward method takes in input tensors and applies Flash Attention.
   - `_flash_attn_uses_top_left_mask`: A flags whether Flash Attention uses top-left aligned causal mask.
   - `attention_mask`, `causal_attention_mask`, and `output_attentions` are optional parameters.

2. **Query, Key, and Value Projection**: The `q_proj`, `k_proj`, and `v_proj` methods project the input tensors to compute query states, key states, and value states.

3. **Dropout Rate**: The `dropout_rate` is calculated from the model configuration or set to 0 during inference.

**Notable Dependencies and Requirements:**

- `torch`: Library for tensor computations.
- Other TensorFlow models (e.g., ` transformers.models.llama.modeling_llama.LlamaFlashAttention2.forward`).


key: chroma:document
string_value:
CODE:

class Multimodal2VisionMLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.activation_fn = ACT2FN[config.hidden_act]
        self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
        self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states = self.fc1(hidden_states)
        hidden_states = self.activation_fn(hidden_states)
        hidden_states = self.fc2(hidden_states)


SUMMARY:
**Multimodal2VisionMLP Class Summary**

* **Main Purpose**: A multimodal fusion neural network that leverages multilayer perceptrons (MLPs) for vision processing.

Primary Functionality:
The `Multimodal2VisionMLP` class implements a multilayer perceptron-based neural network for vision processing, utilizing the provided configuration and activations function.

**Key Implementation Details**:

* **Activation Functions**: Utilizes `ACT2FN[config.hidden_act]` activation function.
* **Layers**: Two fully connected (`fc1` and `fc2`) layers with sizes defined by `config.hidden_size` and `config.intermediate_size`.
* **Forward Pass**: The input tensor is passed through the first linear layer, then undergoes activation, followed by the second linear layer.

**Dependencies/Requirements**:

* References `torch` library for tensor operations.
* Inherents from `nn.Module` class for neural network architecture.


key: chroma:document
string_value:
CODE:

class Multimodal2VisionEncoderLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.embed_dim = config.hidden_size
        self.self_attn = MULTIMODAL2_VISION_ATTENTION_CLASSES[config._attn_implementation](config)
        self.layer_norm1 = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_eps)
        self.mlp = Multimodal2VisionMLP(config)
        self.layer_norm2 = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_eps)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: torch.Tensor,
        causal_attention_mask: torch.Tensor,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.FloatTensor]:
        """
        Args:
            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
            attention_mask (`torch.FloatTensor`): attention mask of size
                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
                `(config.encoder_attention_heads,)`.
            output_attentions (`bool`, *optional*):
                Whether or not to return the attentions tensors of all attention layers. See `attentions` under
                returned tensors for more detail.
        """
        residual = hidden_states

        hidden_states = self.layer_norm1(hidden_states)
        hidden_states, attn_weights = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            causal_attention_mask=causal_attention_mask,
            output_attentions=output_attentions,
        )
        hidden_states = residual + hidden_states

        residual = hidden_states
        hidden_states = self.layer_norm2(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

        outputs = (hidden_states,)

        if output_attentions:
            outputs += (attn_weights,)



SUMMARY:
**Multimodal2VisionEncoderLayer Class Summary**

### Purpose and Functionality

The `Multimodal2VisionEncoderLayer` class is a module in a transformer encoder architecture, designed to process multimodal input data. It takes in sequential data of embedding dimension `config.hidden_size`, applies self-attention mechanisms, and follows up with a feed-forward neural network (FFNN) block.

### Key Implementation Details

* The layer consists of:
	+ Two layer normalization layers (`layer_norm1` and `layer_norm2`) applied to the input data.
	+ A self-attention mechanism (`self_attn`) implemented using `MULTIMODAL2_VISION_ATTENTION_CLASSES[config._attn_implementation]`.
	+ An FFNN block (`mlp`) after the second layer normalization.
* The forward pass involves:
	1. Applying the first self-attention mechanism and layer normalization.
	2. Residual addition of the input data after the first layer normalization.
	3. Repeating steps 1-2 for the second layer normalization and FFNN block.

### Notable Dependencies and Requirements

* `MULTIMODAL2_VISION_ATTENTION_CLASSES[config._attn_implementation]`: a class implementing the attention mechanism based on `config._attn_implicit`.
* Torch library (`torch.Tensor`, `nn.Module`) for building and manipulating neural networks.
* Optional dependency: the `Optional` class from PyTorch for handling boolean inputs.

### Method Signatures and Interfaces

* The primary method of interest is `forward`, which takes in input tensors (`hidden_states`, `attention_mask`, `causal_attention_mask`) and optional inputs (`output_attentions`).


key: chroma:document
string_value:
CODE:

class Multimodal2VisionEncoder(nn.Module):
    """
    Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a
    [`Multimodal2VisionEncoderLayer`].

    Args:
        config: Multimodal2VisionConfig
    """

    def __init__(self, config):
        super().__init__()
        self.config = config
        self.layers = nn.ModuleList([Multimodal2VisionEncoderLayer(config) for _ in range(config.num_hidden_layers)])
        self.gradient_checkpointing = False

    def forward(
        self,
        inputs_embeds,
        attention_mask: Optional[torch.Tensor] = None,
        causal_attention_mask: Optional[torch.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutput]:
        r"""
        Args:
            inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
                Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.
                This is useful if you want more control over how to convert `input_ids` indices into associated vectors
                than the model's internal embedding lookup matrix.
            attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:

                - 1 for tokens that are **not masked**,
                - 0 for tokens that are **masked**.

                [What are attention masks?](../glossary#attention-mask)
            causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Causal mask for the text model. Mask values selected in `[0, 1]`:

                - 1 for tokens that are **not masked**,
                - 0 for tokens that are **masked**.

                [What are attention masks?](../glossary#attention-mask)
            output_attentions (`bool`, *optional*):
                Whether or not to return the attentions tensors of all attention layers. See `attentions` under
                returned tensors for more detail.
            output_hidden_states (`bool`, *optional*):
                Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
                for more detail.
            return_dict (`bool`, *optional*):
                Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        encoder_states = () if output_hidden_states else None
        all_attentions = () if output_attentions else None

        hidden_states = inputs_embeds
        for idx, encoder_layer in enumerate(self.layers):
            if output_hidden_states:
                encoder_states = encoder_states + (hidden_states,)
            if self.gradient_checkpointing and self.training:
                layer_outputs = self._gradient_checkpointing_func(
                    encoder_layer.__call__,
                    hidden_states,
                    attention_mask,
                    causal_attention_mask,
                    output_attentions,
                )
            else:
                layer_outputs = encoder_layer(
                    hidden_states,
                    attention_mask,
                    causal_attention_mask,
                    output_attentions=output_attentions,
                )

            hidden_states = layer_outputs[0]

            if output_attentions:
                all_attentions = all_attentions + (layer_outputs[1],)

        if output_hidden_states:
            encoder_states = encoder_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, encoder_states, all_attentions] if v is not None)
        return BaseModelOutput(
            last_hidden_state=hidden_states, hidden_states=encoder_states, attentions=all_attentions


SUMMARY:
**Summary**

The `Multimodal2VisionEncoder` class is a transformer encoder consisting of multiple attention layers. It takes in input embeddings and an optional batch of attention masks. The primary purpose is to process sequential data using self-attention mechanisms.

**Key Implementation Details**

*   `forward` method: The main entry point for processing input data.
    *   Applies multiple `[Multimodal2VisionEncoderLayer](https://huggingface.co/transformers/_modules Transformers/modeling_multimodal/vision_encoder_layer.html)()` instances in sequence, with optional gradient checkpointing enabled.
*   **`AttentionMask` and `CausalAttentionMask`**: Optional input masks to avoid performing attention on padding token indices.
*   **Output Options**:
    *   `[output_attentions](https://huggingface.co/transformers/_modules Transformers/utils/modeling_utils.html#ModelConfig.get_output_attentions())`: Controls whether to return attentions tensors of all attention layers.
    *   `[output_hidden_states](https://huggingface.co/transformers/_modules Transformers/utils/modeling_utils.html#ModelConfig.get_output_hidden_states())`: Controls whether to return hidden states of all layers.

**Notable Dependencies and Requirements**

*   `nn.ModuleList` and `Multimodal2VisionEncoderLayer` from PyTorch Transformers module.
*   `torch.Tensor` for input embeddings, attention masks, and output tensors.


key: chroma:document
string_value:
CODE:

class Multimodal2VisionEmbeddings(nn.Module):
    def __init__(self, config: Multimodal2VisionConfig):
        super().__init__()
        self.config = config
        self.embed_dim = config.hidden_size
        self.image_size = config.image_size
        self.patch_size = config.patch_size

        self.class_embedding = nn.Parameter(torch.randn(self.embed_dim))

        self.patch_embedding = nn.Conv2d(
            in_channels=config.num_channels,
            out_channels=self.embed_dim,
            kernel_size=self.patch_size,
            stride=self.patch_size,
            bias=False,
        )

        self.num_patches = (self.image_size // self.patch_size) ** 2
        self.num_positions = self.num_patches + 1
        self.position_embedding = nn.Embedding(self.num_positions, self.embed_dim)
        self.register_buffer("position_ids", torch.arange(self.num_positions).expand((1, -1)), persistent=False)

    def interpolate_pos_encoding(self, embeddings: torch.Tensor, height: int, width: int) -> torch.Tensor:
        """
        This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
        images. This method is also adapted to support torch.jit tracing.

        Adapted from:
        - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
        - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
        """

        num_patches = embeddings.shape[1] - 1
        position_embedding = self.position_embedding.weight.unsqueeze(0)
        num_positions = position_embedding.shape[1] - 1

        # always interpolate when tracing to ensure the exported model works for dynamic input shapes
        if not torch.jit.is_tracing() and num_patches == num_positions and height == width:
            return self.position_embedding(self.position_ids)

        class_pos_embed = position_embedding[:, :1]
        patch_pos_embed = position_embedding[:, 1:]

        dim = embeddings.shape[-1]

        new_height = height // self.patch_size
        new_width = width // self.patch_size

        sqrt_num_positions = torch_int(num_positions**0.5)
        patch_pos_embed = patch_pos_embed.reshape(1, sqrt_num_positions, sqrt_num_positions, dim)
        patch_pos_embed = patch_pos_embed.permute(0, 3, 1, 2)

        patch_pos_embed = nn.functional.interpolate(
            patch_pos_embed,
            size=(new_height, new_width),
            mode="bicubic",
            align_corners=False,
        )

        patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim)

        return torch.cat((class_pos_embed, patch_pos_embed), dim=1)

    def forward(self, pixel_values: torch.FloatTensor, interpolate_pos_encoding=False) -> torch.Tensor:
        batch_size, _, height, width = pixel_values.shape
        if not interpolate_pos_encoding and (height != self.image_size or width != self.image_size):
            raise ValueError(
                f"Input image size ({height}*{width}) doesn't match model" f" ({self.image_size}*{self.image_size})."
            )
        target_dtype = self.patch_embedding.weight.dtype
        patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype))  # shape = [*, width, grid, grid]
        patch_embeds = patch_embeds.flatten(2).transpose(1, 2)

        class_embeds = self.class_embedding.expand(batch_size, 1, -1)
        embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
        if interpolate_pos_encoding:
            embeddings = embeddings + self.interpolate_pos_encoding(embeddings, height, width)
        else:
            embeddings = embeddings + self.position_embedding(self.position_ids)


SUMMARY:
**Multimodal2VisionEmbeddings Class Summary**

**Main Purpose and Functionality**

This class is designed for multimodal 2 Vision Embeddings, allowing the model to integrate visual and textual inputs. It maps input pixels to dense embeddings by applying convolutional patches and position encodings.

**Key Implementation Details**

* The `Multimodal2VisionEmbeddings` class maps input pixels to embeddings using a convolutional patch embedding layer (`self.patch_embedding`) and position encoding layer (`self.position_embedding`).
* The `interpolate_pos_encoding` method allows for interpolation of pre-trained position encodings, which enables the model to handle dynamic input shapes.
* The `forward` method combines patch embeddings with class embeddings and optionally applies position or interpolated position encodings.

**Notable Dependencies and Requirements**

* `nn.Module`: serves as the base class for building neural networks
* `torch.randn`, `torch.Embedding`, `nn.Conv2d`, `nn.Parameter`: build various layers
* `torch.jit.is_tracing()` and `torch.functional.interpolate`: require PyTorch tracing support and bicubic interpolation, respectively.

**Notable Interfaces and Class Variables**

* `config` (required): holds configuration settings from the `Multimodal2VisionConfig`
* `self.num_patches`, `self.position_ids`: buffer variables for storing intermediate positions.
* `self.patch_embedding`, `self.class_embedding`, `self.position_embedding`: neural network layers.


key: chroma:document
string_value:
CODE:

class Multimodal2VisionTransformer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        embed_dim = config.hidden_size

        self.embeddings = Multimodal2VisionEmbeddings(config)
        self.pre_layrnorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)
        self.encoder = Multimodal2VisionEncoder(config)
        self.post_layernorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)

    @add_start_docstrings_to_model_forward(MULTIMODAL2_VISION_INPUTS_DOCSTRING)
    @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=Multimodal2VisionConfig)
    def forward(
        self,
        pixel_values: Optional[torch.FloatTensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        interpolate_pos_encoding: Optional[bool] = False,
    ) -> Union[Tuple, BaseModelOutputWithPooling]:
        r"""
        Returns:

        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if pixel_values is None:
            raise ValueError("You have to specify pixel_values")

        hidden_states = self.embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
        hidden_states = self.pre_layrnorm(hidden_states)

        encoder_outputs = self.encoder(
            inputs_embeds=hidden_states,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        last_hidden_state = encoder_outputs[0]
        pooled_output = last_hidden_state[:, 0, :]
        pooled_output = self.post_layernorm(pooled_output)

        if not return_dict:
            return (last_hidden_state, pooled_output) + encoder_outputs[1:]

        return BaseModelOutputWithPooling(
            last_hidden_state=last_hidden_state,
            pooler_output=pooled_output,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,


SUMMARY:
**Multimodal2VisionTransformer Class Summary**

The `Multimodal2VisionTransformer` class is a PyTorch-based transformer model designed for multimodal processing.

* **Primary Purpose**: This model processes video and text inputs from different modalities to generate visual outputs.
* **Key Implementation Details**:
	+ The model consists of an encoding block (`Multimodal2VisionEncoder`) followed by a pooling layer.
	+ The `Multimodal2VisionEmbeddings` component is used to embed input data from both modalities (video and text).
	+ The `pre_layrnorm` and `post_layernorm` layers are applied to the embedded data to normalize it.
* **Notable Dependencies**:
	+ `nn.Module`: Base PyTorch model class.
	+ `Multimodal2VisionEncoder`: Custom encoder component for multimodal processing.
	+ `Multimodal2VisionEmbeddings`: Custom embedding layer for multimodal input data.

**Method Signature Summary**

The `forward` method takes in the following inputs:

* `(pixel_values, ..., interpolate_pos_encoding)`
	+ `pixel_values`: Input video data (torch.FloatTensor).
	+ `output_attentions`, `output_hidden_states`, and `return_dict`: Flags controlling output formats.
	+ `interpolate_pos_encoding`: Flag for position encoding interpolation.

The method returns a tuple containing:

* `(last_hidden_state, pooled_output) + encoder_outputs[1:]` when not returning in a dictionary format.


key: chroma:document
string_value:
CODE:

class Multimodal2VisionPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """

    config_class = Multimodal2Config
    base_model_prefix = "multimodal2_vision"
    supports_gradient_checkpointing = True
    _supports_sdpa = True
    _supports_flash_attn_2 = True

    def _init_weights(self, module):
        """Initialize the weights"""
        if isinstance(module, Multimodal2VisionMLP):


SUMMARY:
**Summary:**

The `Multimodal2VisionPreTrainedModel` class is an abstract base class for pre-trained models. It provides functionality for downloading and loading pretrained models. The primary purpose is to handle weights initialization.

**Key Implementation Details:**

* Initializes weights using the `_init_weights` method.
* Specifically initializes `Multimodal2VisionMLP` modules, but does not perform any operation (pass statement).

**Notable Dependencies or Requirements:**

* No external dependencies are referenced by exact name in this code snippet.
* The following classes/interfaces are implied to be part of the larger architecture:
	+ `PreTrainedModel`
	+ `Multimodal2Config`

Note that without the full context of the `Multimodal2VisionMLP` class, some details remain unclear.


key: chroma:document
string_value:
CODE:

class RobertaEmbeddings(nn.Module):
    """Construct the embeddings from word, position and token_type embeddings."""

    def __init__(self, config):
        super().__init__()
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
        self.position_embeddings = nn.Embedding(
            config.max_position_embeddings, config.hidden_size, config.pad_token_id
        )
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)

        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
        # any TensorFlow checkpoint file
        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        # position_ids (1, len position emb) is contiguous in memory and exported when serialized
        self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")
        self.register_buffer(
            "position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)), persistent=False
        )
        self.register_buffer(
            "token_type_ids", torch.zeros(self.position_ids.size(), dtype=torch.long), persistent=False
        )
        self.pad_token_id = config.pad_token_id

    def forward(
        self,
        input_ids: Optional[torch.LongTensor] = None,
        token_type_ids: Optional[torch.LongTensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        inputs_embeds: Optional[torch.FloatTensor] = None,
        past_key_values_length: int = 0,
    ) -> torch.Tensor:
        if input_ids is not None:
            input_shape = input_ids.size()
        else:
            input_shape = inputs_embeds.size()[:-1]

        seq_length = input_shape[1]

        if position_ids is None:
            position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]

        # Setting the token_type_ids to the registered buffer in constructor where it is all zeros, which usually occurs
        # when its auto-generated, registered buffer helps users when tracing the model without passing token_type_ids, solves
        # issue #5664
        if token_type_ids is None:
            if hasattr(self, "token_type_ids"):
                buffered_token_type_ids = self.token_type_ids[:, :seq_length]
                buffered_token_type_ids_expanded = buffered_token_type_ids.expand(input_shape[0], seq_length)
                token_type_ids = buffered_token_type_ids_expanded
            else:
                token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)

        if inputs_embeds is None:
            inputs_embeds = self.word_embeddings(input_ids)
        token_type_embeddings = self.token_type_embeddings(token_type_ids)

        embeddings = inputs_embeds + token_type_embeddings
        if self.position_embedding_type == "absolute":
            position_embeddings = self.position_embeddings(position_ids)
            embeddings += position_embeddings
        embeddings = self.LayerNorm(embeddings)
        embeddings = self.dropout(embeddings)


SUMMARY:
**Summary:**

The `RobertaEmbeddings` class is a PyTorch implementation of the BERT embeddings layer. Its primary purpose is to construct embeddings from word, position, and token type embeddings.

**Key Implementation Details:**

1. **Module Architecture:** The module consists of:
	* `word_embeddings`: word embedding layer with `config.vocab_size` vocab and `config.hidden_size` hidden dim.
	* `position_embeddings`: position embedding layer with `config.max_position_embeddings` max pos emb and `config.hidden_size` hidden dim.
	* `token_type_embeddings`: token type embedding layer with `config.type_vocab_size` tok vocab and `config.hidden_size` hidden dim.
	* `LayerNorm`, `dropout`, and `position_embedding_type`.
2. **Forward Pass:** The forward pass takes in input IDs, token types, positions, and optional inputs embeds:
	* If input IDs is provided, it uses the word embeddings to get the embeddings.
	* Otherwise, it uses the position embeddings to get the embeddings.
	* It adds the token type embeddings to the result.
	* It applies LayerNorm and dropout to the result.

**Notable Dependencies:**

1. `nn.Module`, `nn.Embedding`, `LayerNorm`, `Dropout`
2. `torch` library

**Interfaces/Inherited Interfaces:**

1. The class inherits from `nn.Module`.


key: chroma:document
string_value:
CODE:

class RobertaSelfAttention(nn.Module):
    def __init__(self, config, position_embedding_type=None):
        super().__init__()
        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
            raise ValueError(
                f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
                f"heads ({config.num_attention_heads})"
            )

        self.num_attention_heads = config.num_attention_heads
        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
        self.all_head_size = self.num_attention_heads * self.attention_head_size

        self.query = nn.Linear(config.hidden_size, self.all_head_size)
        self.key = nn.Linear(config.hidden_size, self.all_head_size)
        self.value = nn.Linear(config.hidden_size, self.all_head_size)

        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
        self.position_embedding_type = position_embedding_type or getattr(
            config, "position_embedding_type", "absolute"
        )
        if self.position_embedding_type == "relative_key" or self.position_embedding_type == "relative_key_query":
            self.max_position_embeddings = config.max_position_embeddings
            self.distance_embedding = nn.Embedding(2 * config.max_position_embeddings - 1, self.attention_head_size)

        self.is_decoder = config.is_decoder

    def transpose_for_scores(self, x: torch.Tensor) -> torch.Tensor:
        new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
        x = x.view(new_x_shape)
        return x.permute(0, 2, 1, 3)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor]:
        mixed_query_layer = self.query(hidden_states)

        # If this is instantiated as a cross-attention module, the keys
        # and values come from an encoder; the attention mask needs to be
        # such that the encoder's padding tokens are not attended to.
        is_cross_attention = encoder_hidden_states is not None

        if is_cross_attention and past_key_value is not None:
            # reuse k,v, cross_attentions
            key_layer = past_key_value[0]
            value_layer = past_key_value[1]
            attention_mask = encoder_attention_mask
        elif is_cross_attention:
            key_layer = self.transpose_for_scores(self.key(encoder_hidden_states))
            value_layer = self.transpose_for_scores(self.value(encoder_hidden_states))
            attention_mask = encoder_attention_mask
        elif past_key_value is not None:
            key_layer = self.transpose_for_scores(self.key(hidden_states))
            value_layer = self.transpose_for_scores(self.value(hidden_states))
            key_layer = torch.cat([past_key_value[0], key_layer], dim=2)
            value_layer = torch.cat([past_key_value[1], value_layer], dim=2)
        else:
            key_layer = self.transpose_for_scores(self.key(hidden_states))
            value_layer = self.transpose_for_scores(self.value(hidden_states))

        query_layer = self.transpose_for_scores(mixed_query_layer)

        use_cache = past_key_value is not None
        if self.is_decoder:
            # if cross_attention save Tuple(torch.Tensor, torch.Tensor) of all cross attention key/value_states.
            # Further calls to cross_attention layer can then reuse all cross-attention
            # key/value_states (first "if" case)
            # if uni-directional self-attention (decoder) save Tuple(torch.Tensor, torch.Tensor) of
            # all previous decoder key/value_states. Further calls to uni-directional self-attention
            # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
            # if encoder bi-directional self-attention `past_key_value` is always `None`
            past_key_value = (key_layer, value_layer)

        # Take the dot product between "query" and "key" to get the raw attention scores.
        attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))

        if self.position_embedding_type == "relative_key" or self.position_embedding_type == "relative_key_query":
            query_length, key_length = query_layer.shape[2], key_layer.shape[2]
            if use_cache:
                position_ids_l = torch.tensor(key_length - 1, dtype=torch.long, device=hidden_states.device).view(
                    -1, 1
                )
            else:
                position_ids_l = torch.arange(query_length, dtype=torch.long, device=hidden_states.device).view(-1, 1)
            position_ids_r = torch.arange(key_length, dtype=torch.long, device=hidden_states.device).view(1, -1)
            distance = position_ids_l - position_ids_r

            positional_embedding = self.distance_embedding(distance + self.max_position_embeddings - 1)
            positional_embedding = positional_embedding.to(dtype=query_layer.dtype)  # fp16 compatibility

            if self.position_embedding_type == "relative_key":
                relative_position_scores = torch.einsum("bhld,lrd->bhlr", query_layer, positional_embedding)
                attention_scores = attention_scores + relative_position_scores
            elif self.position_embedding_type == "relative_key_query":
                relative_position_scores_query = torch.einsum("bhld,lrd->bhlr", query_layer, positional_embedding)
                relative_position_scores_key = torch.einsum("bhrd,lrd->bhlr", key_layer, positional_embedding)
                attention_scores = attention_scores + relative_position_scores_query + relative_position_scores_key

        attention_scores = attention_scores / math.sqrt(self.attention_head_size)
        if attention_mask is not None:
            # Apply the attention mask is (precomputed for all layers in RobertaModel forward() function)
            attention_scores = attention_scores + attention_mask

        # Normalize the attention scores to probabilities.
        attention_probs = nn.functional.softmax(attention_scores, dim=-1)

        # This is actually dropping out entire tokens to attend to, which might
        # seem a bit unusual, but is taken from the original Transformer paper.
        attention_probs = self.dropout(attention_probs)

        # Mask heads if we want to
        if head_mask is not None:
            attention_probs = attention_probs * head_mask

        context_layer = torch.matmul(attention_probs, value_layer)

        context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
        new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
        context_layer = context_layer.view(new_context_layer_shape)

        outputs = (context_layer, attention_probs) if output_attentions else (context_layer,)

        if self.is_decoder:
            outputs = outputs + (past_key_value,)


SUMMARY:
**Overview**

The `RobertaSelfAttention` class implements a self-attention mechanism in the context of transformer-based neural networks. It is designed to compute the weighted sum of input elements that are most relevant to a given element, based on their similarity.

**Key Functionality**

1.  Initializes the attention head sizes and linear layers for query, key, and value.
2.  Transposes input tensors for computation efficiency.
3.  Computes attention scores by taking the dot product between query and key representations.
4.  Applies position embeddings to consider relative positions between tokens.
5.  Normalizes attention scores to produce probabilities.
6.  Attenuates self-attention weights through dropout and applies masked heads.

**Notable Components**

*   `query`, `key`, and `value` linear layers
*   `transpose_for_scores` method for efficient computation
*   `position_embedding_type` parameter controlling relative position embedding usage

**Dependencies**

*   PyTorch (`torch`)
*   Math library (math)

**Key Interfaces/Methods**

*   `__init__(config, position_embedding_type)`
*   `transpose_for_scores(x)`
*   `forward(hidden_states, *, attention_mask, head_mask, encoder_hidden_states, ..., output_attentions)`


key: chroma:document
string_value:
CODE:

class RobertaSdpaSelfAttention(RobertaSelfAttention):
    def __init__(self, config, position_embedding_type=None):
        super().__init__(config, position_embedding_type=position_embedding_type)
        self.dropout_prob = config.attention_probs_dropout_prob
        self.require_contiguous_qkv = version.parse(get_torch_version()) < version.parse("2.2.0")

    # Adapted from RobertaSelfAttention
    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor]:
        if self.position_embedding_type != "absolute" or output_attentions or head_mask is not None:
            # TODO: Improve this warning with e.g. `model.config._attn_implementation = "manual"` once implemented.
            logger.warning_once(
                "RobertaSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support "
                "non-absolute `position_embedding_type` or `output_attentions=True` or `head_mask`. Falling back to "
                "the manual attention implementation, but specifying the manual implementation will be required from "
                "Transformers version v5.0.0 onwards. This warning can be removed using the argument "
                '`attn_implementation="eager"` when loading the model.'
            )
            return super().forward(
                hidden_states,
                attention_mask,
                head_mask,
                encoder_hidden_states,
                encoder_attention_mask,
                past_key_value,
                output_attentions,
            )

        bsz, tgt_len, _ = hidden_states.size()

        query_layer = self.transpose_for_scores(self.query(hidden_states))

        # If this is instantiated as a cross-attention module, the keys and values come from an encoder; the attention
        # mask needs to be such that the encoder's padding tokens are not attended to.
        is_cross_attention = encoder_hidden_states is not None

        current_states = encoder_hidden_states if is_cross_attention else hidden_states
        attention_mask = encoder_attention_mask if is_cross_attention else attention_mask

        # Check `seq_length` of `past_key_value` == `len(current_states)` to support prefix tuning
        if is_cross_attention and past_key_value and past_key_value[0].shape[2] == current_states.shape[1]:
            key_layer, value_layer = past_key_value
        else:
            key_layer = self.transpose_for_scores(self.key(current_states))
            value_layer = self.transpose_for_scores(self.value(current_states))
            if past_key_value is not None and not is_cross_attention:
                key_layer = torch.cat([past_key_value[0], key_layer], dim=2)
                value_layer = torch.cat([past_key_value[1], value_layer], dim=2)

        if self.is_decoder:
            # if cross_attention save Tuple(torch.Tensor, torch.Tensor) of all cross attention key/value_states.
            # Further calls to cross_attention layer can then reuse all cross-attention
            # key/value_states (first "if" case)
            # if uni-directional self-attention (decoder) save Tuple(torch.Tensor, torch.Tensor) of
            # all previous decoder key/value_states. Further calls to uni-directional self-attention
            # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
            # if encoder bi-directional self-attention `past_key_value` is always `None`
            past_key_value = (key_layer, value_layer)

        # SDPA with memory-efficient backend is broken in torch==2.1.2 when using non-contiguous inputs and a custom
        # attn_mask, so we need to call `.contiguous()` here. This was fixed in torch==2.2.0.
        # Reference: https://github.com/pytorch/pytorch/issues/112577
        if self.require_contiguous_qkv and query_layer.device.type == "cuda" and attention_mask is not None:
            query_layer = query_layer.contiguous()
            key_layer = key_layer.contiguous()
            value_layer = value_layer.contiguous()

        # We dispatch to SDPA's Flash Attention or Efficient kernels via this `is_causal` if statement instead of an inline conditional assignment
        # in SDPA to support both torch.compile's dynamic shapes and full graph options. An inline conditional prevents dynamic shapes from compiling.
        # The tgt_len > 1 is necessary to match with AttentionMaskConverter.to_causal_4d that does not create
        # a causal mask in case tgt_len == 1.
        is_causal = (
            True if self.is_decoder and not is_cross_attention and attention_mask is None and tgt_len > 1 else False
        )

        attn_output = torch.nn.functional.scaled_dot_product_attention(
            query_layer,
            key_layer,
            value_layer,
            attn_mask=attention_mask,
            dropout_p=self.dropout_prob if self.training else 0.0,
            is_causal=is_causal,
        )

        attn_output = attn_output.transpose(1, 2)
        attn_output = attn_output.reshape(bsz, tgt_len, self.all_head_size)

        outputs = (attn_output,)
        if self.is_decoder:
            outputs = outputs + (past_key_value,)


SUMMARY:
**RobertaSdpaSelfAttention Class Summary**

The `RobertabaSdpaSelfAttention` class is a variant of the standard RoBerta self-attention mechanism.

**Main Purpose and Functionality**

This class implements a modified version of the scaled dot-product attention (SDPA) that uses a memory-efficient backend.
It dispatches to either Flash Attention or Efficient kernels depending on the specific configuration.
The class also provides an ability to use `past_key_value` for prefix tuning.

**Key Implementation Details:**

* The `forward` method takes in various inputs and returns the attention output, query layers, value layers, key layers, and value layers.
* It uses the `torch.nn.functional.scaled_dot_product_attention` function under the hood.
* The `require_contiguous_qkv` attribute control the behavior of the memory-efficient backend.

**Notable Dependencies or Requirements:**

* `torch` with version 2.0.0 or later for optimized performance.
* `RobertaSelfAttention` as a base class.

Here is an example of how to use this class:
```python
import torch

config = ...  # define the model configuration
attention_model = RobertaSdpaSelfAttention(config)
```
**Warning:**

The class uses warnings when non-absolute position embeddings or output attentions are used, indicating that it will fall back to a manual attention implementation.
This can be improved by specifying the `attn_implementation` argument to `"eager"` when loading the model.


key: chroma:document
string_value:
CODE:

class RobertaSelfOutput(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        hidden_states = self.LayerNorm(hidden_states + input_tensor)


SUMMARY:
**Concise Summary**

This code defines a `RobertaSelfOutput` class (`nn.Module`) that extends PyTorch's neural network functionality.

**Main Purpose and Functionality**
The primary purpose of this class is to calculate the self-attention output for a given input tensor using the Roberta model architecture. It applies density, dropout, and layer normalization operations to transform the hidden states into a final output tensor.

**Key Implementation Details**

* `[ Dense Layer ]`: A linear (`nn.Linear`) transformation with `config.hidden_size` as input and output dimensions.
* `[ Dropout Layer ]`: Applies random dropout to the transformed hidden state with probability `config.hidden_dropout_prob`.
* `[ LayerNorm Layer ]`: Applies layer normalization (additive) to the combined output of density and dropout layers, using eps-value from the `LayerNorm` arguments.

**Notable Dependencies and Requirements**

* `torch.nn` (PyTorch's neural network module)
* `nn.Linear`, `nn.LayerNorm`, `nn.Dropout` classes from PyTorch (`torch`)


key: chroma:document
string_value:
CODE:

class RobertaAttention(nn.Module):
    def __init__(self, config, position_embedding_type=None):
        super().__init__()
        self.self = ROBERTA_SELF_ATTENTION_CLASSES[config._attn_implementation](
            config, position_embedding_type=position_embedding_type
        )
        self.output = RobertaSelfOutput(config)
        self.pruned_heads = set()

    def prune_heads(self, heads):
        if len(heads) == 0:
            return
        heads, index = find_pruneable_heads_and_indices(
            heads, self.self.num_attention_heads, self.self.attention_head_size, self.pruned_heads
        )

        # Prune linear layers
        self.self.query = prune_linear_layer(self.self.query, index)
        self.self.key = prune_linear_layer(self.self.key, index)
        self.self.value = prune_linear_layer(self.self.value, index)
        self.output.dense = prune_linear_layer(self.output.dense, index, dim=1)

        # Update hyper params and store pruned heads
        self.self.num_attention_heads = self.self.num_attention_heads - len(heads)
        self.self.all_head_size = self.self.attention_head_size * self.self.num_attention_heads
        self.pruned_heads = self.pruned_heads.union(heads)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor]:
        self_outputs = self.self(
            hidden_states,
            attention_mask,
            head_mask,
            encoder_hidden_states,
            encoder_attention_mask,
            past_key_value,
            output_attentions,
        )
        attention_output = self.output(self_outputs[0], hidden_states)
        outputs = (attention_output,) + self_outputs[1:]  # add attentions if we output them


SUMMARY:
**Concise Summary**

The `RobertaAttention` class implements the multi-head attention mechanism using the RoBERTa architecture. It inherits from PyTorch's `nn.Module`.

**Main Purpose and Functionality**
---------------------------

* Attentive computation of weighted sums with learned query, key, and value vectors.
* Allows for dynamic pruning of irrelevant attention heads.

**Key Implementation Details**
-----------------------------

* Employs the ROBERTA_SELF_ATTENTION_CLASSES implementation from config.
* Utilizes `find_pruneable_heads_and_indices` to identify pruneable heads.
* Modifies linear layers using `prune_linear_layer` after identifying pruneable heads.

**Notable Dependencies and Requirements**
-----------------------------------------

* Extends PyTorch's `nn.Module`.
* Requires `ROBERTA_SELF_ATTENTION_CLASSES`, `prune_linear_layer`, `find_pruneable_heads_and_indices`, and `config._attn_implementation`.

**Method Signatures and Interfaces**
--------------------------------------

* `__init__: self, config: nn(cfg), position_embedding_type=None`
* `prune_heads: heads (list) -> None`
* `forward: hidden_states (torch.Tensor), *args (Optional[torch.FloatTensor] = None)`


key: chroma:document
string_value:
CODE:

class RobertaIntermediate(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
        if isinstance(config.hidden_act, str):
            self.intermediate_act_fn = ACT2FN[config.hidden_act]
        else:
            self.intermediate_act_fn = config.hidden_act

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.intermediate_act_fn(hidden_states)


SUMMARY:
**Summary**

The `RobertaIntermediate` class is a PyTorch neural network module that implements the attention intermediate layer of the BERT (Bidirectional Encoder Representations from Transformers) architecture. Its primary purpose is to apply an activation function (`config.hidden_act`) to the output of a dense linear transformation (`dense`) of input `hidden_states`.

**Key Features**

* Applies a linear transformation using `nn.Linear`
* Conditional application of an activation function:
	+ If `config.hidden_act` is a string, uses the corresponding mapping from `ACT2FN`
	+ Otherwise, uses the provided `config.hidden_act` value as the activation function
* Inheriting interface: `nn.Module`

**Dependences**

* PyTorch library (`torch`)
* `nn.Linear`, `torch.Tensor`, and other PyTorch modules.


key: chroma:document
string_value:
CODE:

class RobertaOutput(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        hidden_states = self.LayerNorm(hidden_states + input_tensor)


SUMMARY:
**Concise Summary**

The `RobertaOutput` class is an instance of the PyTorch neural network module (`nn.Module`). It serves as the output layer for a transformer-based model. The primary purpose of this class is to apply a dense output layer, followed by normalization and dropout to process input tensor.

**Key Implementation Details**

* **forward method**: Applies self-dense, then dropout on `hidden_states`, and finally layers normalization with added `input_tensor`.
* **Attributes**:
	+ `dense`: A linear layer (`nn.Linear`) applied to `hidden_states` and `input_tensor`.
	+ `LayerNorm`: Layer normalized output of hidden tensor and input tensor.
	+ `dropout`: Applicative dropout on output.

**Notable Dependencies/Requirements**

* Imported classes: `nn.Module`, `nn.Linear`, `nn.LayerNorm`, `nn.Dropout`
* External dependencies:
	+ `torch.Tensor`


key: chroma:document
string_value:
CODE:

class RobertaLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.chunk_size_feed_forward = config.chunk_size_feed_forward
        self.seq_len_dim = 1
        self.attention = RobertaAttention(config)
        self.is_decoder = config.is_decoder
        self.add_cross_attention = config.add_cross_attention
        if self.add_cross_attention:
            if not self.is_decoder:
                raise ValueError(f"{self} should be used as a decoder model if cross attention is added")
            self.crossattention = RobertaAttention(config, position_embedding_type="absolute")
        self.intermediate = RobertaIntermediate(config)
        self.output = RobertaOutput(config)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor]:
        # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
        self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
        self_attention_outputs = self.attention(
            hidden_states,
            attention_mask,
            head_mask,
            output_attentions=output_attentions,
            past_key_value=self_attn_past_key_value,
        )
        attention_output = self_attention_outputs[0]

        # if decoder, the last output is tuple of self-attn cache
        if self.is_decoder:
            outputs = self_attention_outputs[1:-1]
            present_key_value = self_attention_outputs[-1]
        else:
            outputs = self_attention_outputs[1:]  # add self attentions if we output attention weights

        cross_attn_present_key_value = None
        if self.is_decoder and encoder_hidden_states is not None:
            if not hasattr(self, "crossattention"):
                raise ValueError(
                    f"If `encoder_hidden_states` are passed, {self} has to be instantiated with cross-attention layers"
                    " by setting `config.add_cross_attention=True`"
                )

            # cross_attn cached key/values tuple is at positions 3,4 of past_key_value tuple
            cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
            cross_attention_outputs = self.crossattention(
                attention_output,
                attention_mask,
                head_mask,
                encoder_hidden_states,
                encoder_attention_mask,
                cross_attn_past_key_value,
                output_attentions,
            )
            attention_output = cross_attention_outputs[0]
            outputs = outputs + cross_attention_outputs[1:-1]  # add cross attentions if we output attention weights

            # add cross-attn cache to positions 3,4 of present_key_value tuple
            cross_attn_present_key_value = cross_attention_outputs[-1]
            present_key_value = present_key_value + cross_attn_present_key_value

        layer_output = apply_chunking_to_forward(
            self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output
        )
        outputs = (layer_output,) + outputs

        # if decoder, return the attn key/values as the last output
        if self.is_decoder:
            outputs = outputs + (present_key_value,)

        return outputs

    def feed_forward_chunk(self, attention_output):
        intermediate_output = self.intermediate(attention_output)
        layer_output = self.output(intermediate_output, attention_output)


SUMMARY:
**Summary**

The `RobertaLayer` class is a custom PyTorch nn.Module implementing the Roberta transformer layer. It consists of:

1. **Initialization**: Sets default parameters and instantiates attention, intermediate, and output layers based on the provided configuration (`config`).
2. **Forward Pass**:
	* Computes self-attention outputs using `RoBERTaAttention` with optional cross-attention if `add_cross_attention` is enabled.
	* Applies a feed-forward chunk to the self-attention outputs.
3. **Notable Dependencies/Requirements**:
	+ `config`: Provides configuration settings for the layer.
	+ `torch.nn.Module`, `torch.Tensor`
	+ `apply_chunking_to_forward`: A custom function that applies chunking to compute forward pass.

**Key Method Signatures and Interfaces**

* `__init__(config)`: Initializes the layer with the provided configuration.
* `forward(hidden_states: torch.Tensor, ... -> Tuple[torch.Tensor])`: Computes self-attention outputs and applies a feed-forward chunk to produce the final output.
* `RoBERTaAttention(config)`: An instance of the Roberta attention class used in the forward pass.
* `RoBERTaIntermediate(config)`: An instance of the Roberta intermediate layer used in the feed-forward chunk.

**External Dependencies**

* None (self-contained PyTorch code)

**Class Variables**

* `chunk_size_feed_forward`: A configuration setting for the feed-forward chunk size.
* `is_decoder` and `add_cross_attention`: Configuration flags enabling cross-attention.
* `seq_len_dim`, `past_key_value`, and `encoder_hidden_states`: Miscellaneous variables used in the forward pass.


key: chroma:document
string_value:
CODE:

class RobertaEncoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.layer = nn.ModuleList([RobertaLayer(config) for _ in range(config.num_hidden_layers)])
        self.gradient_checkpointing = False

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.FloatTensor] = None,
        head_mask: Optional[torch.FloatTensor] = None,
        encoder_hidden_states: Optional[torch.FloatTensor] = None,
        encoder_attention_mask: Optional[torch.FloatTensor] = None,
        past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = False,
        output_hidden_states: Optional[bool] = False,
        return_dict: Optional[bool] = True,
    ) -> Union[Tuple[torch.Tensor], BaseModelOutputWithPastAndCrossAttentions]:
        all_hidden_states = () if output_hidden_states else None
        all_self_attentions = () if output_attentions else None
        all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None

        if self.gradient_checkpointing and self.training:
            if use_cache:
                logger.warning_once(
                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
                )
                use_cache = False

        next_decoder_cache = () if use_cache else None
        for i, layer_module in enumerate(self.layer):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            layer_head_mask = head_mask[i] if head_mask is not None else None
            past_key_value = past_key_values[i] if past_key_values is not None else None

            if self.gradient_checkpointing and self.training:
                layer_outputs = self._gradient_checkpointing_func(
                    layer_module.__call__,
                    hidden_states,
                    attention_mask,
                    layer_head_mask,
                    encoder_hidden_states,
                    encoder_attention_mask,
                    past_key_value,
                    output_attentions,
                )
            else:
                layer_outputs = layer_module(
                    hidden_states,
                    attention_mask,
                    layer_head_mask,
                    encoder_hidden_states,
                    encoder_attention_mask,
                    past_key_value,
                    output_attentions,
                )

            hidden_states = layer_outputs[0]
            if use_cache:
                next_decoder_cache += (layer_outputs[-1],)
            if output_attentions:
                all_self_attentions = all_self_attentions + (layer_outputs[1],)
                if self.config.add_cross_attention:
                    all_cross_attentions = all_cross_attentions + (layer_outputs[2],)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(
                v
                for v in [
                    hidden_states,
                    next_decoder_cache,
                    all_hidden_states,
                    all_self_attentions,
                    all_cross_attentions,
                ]
                if v is not None
            )
        return BaseModelOutputWithPastAndCrossAttentions(
            last_hidden_state=hidden_states,
            past_key_values=next_decoder_cache,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
            cross_attentions=all_cross_attentions,


SUMMARY:
**Summary:**

This is an implementation of the `RobertaEncoder` class, which inherits from PyTorch's `nn.Module`. The class processes input sequences using multi-head attention and feed-forward networks.

**Primary Purpose:**
The primary purpose of this class is to encode input sequences into a context-free representation that can be used for downstream tasks such as classification or question answering.

**Key Implementation Details:**

*   Multi-layer architecture using `RobertaLayer` modules, which process input sequences in parallel.
*   Support for gradient checkpointing and caching of attention outputs.
*   Optional output of attentions, hidden states, and past key values.
*   Interface with other models for multi-task training.

**Notable Dependencies and Requirements:**

*   `torch` library for tensor operations
*   `nn.ModuleList` to manage multiple layers
*   `RobertaLayer` to implement individual layer logic

**Interfaces Implemented:**

*   Inherited `nn.Module`
*   Supports output formats including `BaseModelOutputWithPastAndCrossAttentions`

This summary aims to capture the core functionality and key features of the provided code chunk, providing a concise overview for developers familiar with PyTorch models.


key: chroma:document
string_value:
CODE:

class RobertaPooler(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.activation = nn.Tanh()

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.
        first_token_tensor = hidden_states[:, 0]
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)


SUMMARY:
**Summary:**

This is a PyTorch implementation of `RobertaPooler` class, which inherits from the `nn.Module` class. The primary purpose of this class is to "pool" the input model by taking only the first token's hidden state.

**Key Details:**

* Takes in a `hidden_states` tensor as input.
* Uses a linear layer (`self.dense`) with dimensions `(config.hidden_size, config.hidden_size)` to transform the first token into pooled output.
* Applies the `torch.nn.Tanh()` activation function to the pooled output.

**Notable Dependencies/Requirements:**

* PyTorch `nn.Module` class
* PyTorch `Linear` and `Tanh` modules (`import torch.nn as nn`)


key: chroma:document
string_value:
CODE:

def load_tf_weights_in_roberta(model, config, tf_checkpoint_path):
    """Load tf checkpoints in a pytorch model."""
    try:
        import re

        import numpy as np
        import tensorflow as tf
    except ImportError:
        logger.error(
            "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see "
            "https://www.tensorflow.org/install/ for installation instructions."
        )
        raise
    tf_path = os.path.abspath(tf_checkpoint_path)
    logger.info(f"Converting TensorFlow checkpoint from {tf_path}")
    # Load weights from TF model
    init_vars = tf.train.list_variables(tf_path)
    names = []
    arrays = []
    for name, shape in init_vars:
        logger.info(f"Loading TF weight {name} with shape {shape}")
        array = tf.train.load_variable(tf_path, name)
        names.append(name)
        arrays.append(array)

    for name, array in zip(names, arrays):
        name = name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(
            n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
            for n in name
        ):
            logger.info(f"Skipping {'/'.join(name)}")
            continue
        pointer = model
        for m_name in name:
            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
                scope_names = re.split(r"_(\d+)", m_name)
            else:
                scope_names = [m_name]
            if scope_names[0] == "kernel" or scope_names[0] == "gamma":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
                pointer = getattr(pointer, "bias")
            elif scope_names[0] == "output_weights":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "squad":
                pointer = getattr(pointer, "classifier")
            else:
                try:
                    pointer = getattr(pointer, scope_names[0])
                except AttributeError:
                    logger.info(f"Skipping {'/'.join(name)}")
                    continue
            if len(scope_names) >= 2:
                num = int(scope_names[1])
                pointer = pointer[num]
        if m_name[-11:] == "_embeddings":
            pointer = getattr(pointer, "weight")
        elif m_name == "kernel":
            array = np.transpose(array)
        try:
            if pointer.shape != array.shape:
                raise ValueError(f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched")
        except ValueError as e:
            e.args += (pointer.shape, array.shape)
            raise
        logger.info(f"Initialize PyTorch weight {name}")
        pointer.data = torch.from_numpy(array)


SUMMARY:
**Function Summary:**

`load_tf_weights_in_roberta`

Loads TensorFlow checkpoints into a PyTorch RoBERTa model.

**Purpose and Functionality:**

This function loads the pre-trained weights from a specified TensorFlow checkpoint file (`tf_checkpoint_path`) and initializes the corresponding variables in the provided PyTorch model instance (`model`). It filters out unnecessary variables, such as Adam optimizer variables and biases, to ensure proper compatibility with PyTorch. The loaded weights are then stored in the model's tensor fields.

**Key Implementation Details:**

* Loads `init_vars` from the TensorFlow checkpoint file
* Iterates through each variable `name` and its corresponding value `array`
* Filters out unnecessary variables based on name patterns
* Assigns the loaded array to the corresponding PyTorch variable using ` pointer.data = torch.from_numpy(array)`

**Notable Dependencies and Requirements:**

* NumPy (`numpy`)
* TensorFlow (`tensorflow`)
* PyTorch (`torch`)

**Interface and Method Signatures:**

* `tf.train.list_variables(tf_path)`: Lists TF variables in the specified path
* `tf.train.load_variable(tf_path, name)`: Loads a single variable from the TensorFlow checkpoint file
* `getattr(model, name)`: Accesses an attribute by its name within the model instance

**External Dependencies:**

* Re (regular expression)
* Logger ( error and info logging functions)

Note: This summary provides a high-level overview of the function's purpose, key implementation details, and notable dependencies. The original code remains unchanged to preserve its integrity.


key: chroma:document
string_value:
CODE:

class RobertaPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """

    config_class = RobertaConfig
    load_tf_weights = load_tf_weights_in_roberta
    base_model_prefix = "roberta"
    supports_gradient_checkpointing = True
    _supports_sdpa = True

    def _init_weights(self, module):
        """Initialize the weights"""
        if isinstance(module, nn.Linear):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
            if module.bias is not None:
                module.bias.data.zero_()
        elif isinstance(module, nn.Embedding):
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
            if module.padding_idx is not None:
                module.weight.data[module.padding_idx].zero_()
        elif isinstance(module, nn.LayerNorm):
            module.bias.data.zero_()


SUMMARY:
**Concise Summary**

### Model: Roberta Pre-Trained Model
```
An abstract base class for handling weights initialization and downloading/preloading pretrained models.

#### Key Highlights:

* Initializes weights using `normal_(mean=0.0, std=config.initializer_range)` for:
	+ `Linear` layers
* Zeroes biases with `zero_()` if present
* Normalizes weight matrix with `fill_(1.0)` in `LayerNorm` layers

#### Configuration:

* `config_class`: RobertaConfig
```

### Notes

No notable dependencies or implementation details are listed, assuming this code is part of a larger package with existing config implementation.

Inherited Interfaces:

* `PreTrainedModel`
* No external dependencies mentioned


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_from_uppercase_model.py


key: chroma:document
string_value:
CODE:
# Check if we can correctly grab dependencies with correct naming from all UPPERCASE old model
class FromUppercaseModelEncoderLayer(CLIPEncoderLayer):


SUMMARY:
**Summary**

This class, `FromUppercaseModelEncoderLayer`, inherits from `CLIPEncoderLayer`. It serves as a transformer layer in the Hugging Face model encoder architecture. The class is responsible for converting input data to uppercase.

**Key Implementation Details**

*   **Primary Purpose:** Converts input data to uppercase.
*   **Method Signatures:**
    *   `__init__(self)`: Initializes the layer with no dependencies specified.
*   **Important Interface:** Inherited from `CLIPEncoderLayer`, which suggests it's an encoder layer in a transformer architecture.

**Notable Dependencies and Requirements**

The class does not appear to have any external dependencies or requirements. However, it inherits interfaces and classes from the parent class `CLIPEncoderLayer`, indicating that it relies on the underlying architecture provided by this class or its immediate parents. 

**Security Note:** Although not explicitly mentioned in this snippet alone, consider that encoding inputs strictly in uppercase might not be suitable for all applications where model input handling requires case sensitivity (for example, during testing or when interacting with APIs).


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/configuration_my_new_model2.py


key: chroma:document
string_value:
CODE:

class MyNewModel2Config(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`GemmaModel`]. It is used to instantiate an Gemma
    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
    defaults will yield a similar configuration to that of the Gemma-7B.
    e.g. [google/gemma-7b](https://huggingface.co/google/gemma-7b)
    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.
    Args:
        vocab_size (`int`, *optional*, defaults to 256000):
            Vocabulary size of the Gemma model. Defines the number of different tokens that can be represented by the
            `inputs_ids` passed when calling [`GemmaModel`]
    ```python
    >>> from transformers import GemmaModel, GemmaConfig
    >>> # Initializing a Gemma gemma-7b style configuration
    >>> configuration = GemmaConfig()
    >>> # Initializing a model from the gemma-7b style configuration
    >>> model = GemmaModel(configuration)
    >>> # Accessing the model configuration
    >>> configuration = model.config
    ```"""

    model_type = "my_new_model2"
    keys_to_ignore_at_inference = ["past_key_values"]
    # Default tensor parallel plan for base model `MyNewModel2Model`
    base_model_tp_plan = {
        "layers.*.self_attn.q_proj": "colwise",
        "layers.*.self_attn.k_proj": "colwise",
        "layers.*.self_attn.v_proj": "colwise",
        "layers.*.self_attn.o_proj": "rowwise",
        "layers.*.mlp.gate_proj": "colwise",
        "layers.*.mlp.up_proj": "colwise",
        "layers.*.mlp.down_proj": "rowwise",
    }

    def __init__(
        self,
        vocab_size=32000,
        hidden_size=4096,
        intermediate_size=11008,
        num_hidden_layers=32,
        num_attention_heads=32,
        num_key_value_heads=None,
        hidden_act="silu",
        max_position_embeddings=2048,
        initializer_range=0.02,
        rms_norm_eps=1e-6,
        use_cache=True,
        pad_token_id=None,
        bos_token_id=1,
        eos_token_id=2,
        pretraining_tp=1,
        tie_word_embeddings=False,
        rope_theta=10000.0,
        rope_scaling=None,
        attention_bias=False,
        attention_dropout=0.0,
        mlp_bias=False,
        head_dim=None,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads

        # for backward compatibility
        if num_key_value_heads is None:
            num_key_value_heads = num_attention_heads

        self.num_key_value_heads = num_key_value_heads
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.rms_norm_eps = rms_norm_eps
        self.pretraining_tp = pretraining_tp
        self.use_cache = use_cache
        self.rope_theta = rope_theta
        self.rope_scaling = rope_scaling
        self.attention_bias = attention_bias
        self.attention_dropout = attention_dropout
        self.mlp_bias = mlp_bias
        self.head_dim = head_dim if head_dim is not None else self.hidden_size // self.num_attention_heads
        # Validate the correctness of rotary position embeddings parameters
        # BC: if there is a 'type' field, copy it it to 'rope_type'.
        if self.rope_scaling is not None and "type" in self.rope_scaling:
            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
        rope_config_validation(self)

        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,


SUMMARY:
**Summary:**

The `MyNewModel2Config` class is a configuration for the Gemma model, inheriting from `PretrainedConfig`. It stores model architecture definitions and has methods to control output shapes.

**Key Implementation Details:**

* Initializes with `vocab_size`, defining token representation capacity
* Sets tensor parallel plan for base model (`MyNewModel2Model`) using `base_model_tp_plan` dictionary
* Sets various model hyperparameters, such as hidden size, intermediate size, and number of attention heads

**Notable Dependencies or Requirements:**

* Inherited from `PretrainedConfig`
* External dependencies:
	+ `gemma-7b` for default configuration
* References related classes:
	+ `GemmaModel`
	+ `MyNewModel2Model`

Overall, this class configures the Gemma model and defines its architecture, including hyperparameters and tensor parallel plan.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_new_task_model.py


key: chroma:document
string_value:
CODE:

class NewTaskModelForNewTask(PaliGemmaForConditionalGeneration):
    main_input_name: ClassVar[str] = "doc_input_ids"  # transformers-related

    def __init__(self, config):
        super().__init__(config=config)

        self.embedding_dim = self.config.embedding_dim
        self.custom_text_proj = nn.Linear(self.config.text_config.hidden_size, self.embedding_dim)

        if self.language_model._tied_weights_keys is not None:
            self._tied_weights_keys = [f"model.language_model.{k}" for k in self.language_model._tied_weights_keys]

        self.post_init()

    def forward(
        self,
        input_ids: torch.LongTensor = None,
        pixel_values: torch.FloatTensor = None,
        attention_mask: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_values: Optional[Union[List[torch.FloatTensor], Cache]] = None,
        token_type_ids: Optional[torch.LongTensor] = None,
        cache_position: Optional[torch.LongTensor] = None,
        inputs_embeds: Optional[torch.FloatTensor] = None,
        labels: Optional[torch.LongTensor] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        num_logits_to_keep: int = 0,
    ):
        r"""
        Returns:
        """
        vlm_outputs = super().forward(
            input_ids=input_ids,
            pixel_values=pixel_values,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_values=past_key_values,
            token_type_ids=token_type_ids,
            cache_position=cache_position,
            inputs_embeds=inputs_embeds,
            labels=labels,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=True,
            return_dict=True,
            num_logits_to_keep=num_logits_to_keep,
        )
        last_hidden_states = vlm_outputs.hidden_states[-1]  # (batch_size, sequence_length, hidden_size)
        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

        # L2 normalization
        embeddings = proj / proj.norm(dim=-1, keepdim=True)  # (batch_size, sequence_length, dim)

        embeddings = embeddings * attention_mask.unsqueeze(-1)  # (batch_size, sequence_length, dim)

        return (embeddings,) + vlm_outputs

    def resize_token_embeddings(
        self,
        new_num_tokens: Optional[int] = None,
        pad_to_multiple_of=None,
    ) -> nn.Embedding:
        model_embeds = self.language_model.resize_token_embeddings(new_num_tokens, pad_to_multiple_of)

        # Update vocab size
        self.config.text_config.vocab_size = model_embeds.num_embeddings
        self.config.vocab_size = model_embeds.num_embeddings
        self.vocab_size = model_embeds.num_embeddings



SUMMARY:
**NewTaskModelForNewTask Class Summary**

**Class Overview**
-----------------

`NewTaskModelForNewTask` is a custom PyTorch model for conditional generation, inheriting from `PaliGemmaForConditionalGeneration`. Its primary purpose is to process input sequences and generate text outputs.

**Key Functionality**
--------------------

The main functionality of this class involves:

*   Processing input sequences with customizable projection
*   Performing L2 normalization on projected sequences
*   Scaling projections by attention masks

**Notable Implementation Details**
---------------------------------

*   Uses `nn.Linear` for custom text projection (`self.custom_text_proj`)
*   Includes a call to `super().forward()` to delegate most processing tasks
*   Performs forward pass with input variables such as `input_ids`, `pixel_values`, and `attention_mask`

**Dependencies and Requirements**
-------------------------------

*   Inheriting from `PaliGemmaForConditionalGeneration`
*   Uses `torch` for computations
*   Assumes availability of `self.language_model` with corresponding attributes

**External Dependencies**
----------------------

*   `PyTorch` library (not explicitly listed but assumed as part of PyTorch-E engine)


key: chroma:document
string_value:
CODE:

class Multimodal1TextRMSNorm(nn.Module):
    def __init__(self, hidden_size, eps=1e-6):
        """
        Multimodal1TextRMSNorm is equivalent to T5LayerNorm
        """
        super().__init__()
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.variance_epsilon = eps

    def forward(self, hidden_states):
        input_dtype = hidden_states.dtype
        hidden_states = hidden_states.to(torch.float32)
        variance = hidden_states.pow(2).mean(-1, keepdim=True)
        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
        return self.weight * hidden_states.to(input_dtype)

    def extra_repr(self):


SUMMARY:
**Class Summary: Multimodal1TextRMSNorm**

**Main Purpose and Functionality**
---------------------------

The `Multimodal1TextRMSNorm` class implements a layer normalization technique, specifically RMSNorm, for multimodal text inputs. It is equivalent to the transformer layer normalization in T5.

* Purpose: Normalize input data by scaling it based on estimated variance.
* Functionality: Applies layer normalization using the RMSNorm algorithm, which scales input data according to an estimated variance calculated from previous steps in the model pipeline.

**Key Implementation Details**
-----------------------------

* **Algorithm:** Uses Root Mean Square (RMS) normalization.
* **Method Signatures:**
	+ `forward(hidden_states):` Takes in normalized hidden states and returns output with weight scaling applied.
	+ `extra_repr():` Returns extra representation details, including model weights shape and epsilon value.

**Dependencies and Requirements**
--------------------------------

* **Inherited Interface:** `nn.Module`
* **External Dependencies:**
	+ `torch`: Used for tensor operations (e.g., `rsqrt`, `pow`, `mean`)
	+ `nn.Parameter`, `torch.ones`: Used to initialize model parameters.
* **Notable Classes and Functions:**
	+ The RMSNorm algorithm relies on the `numpy` library, although not explicitly imported in this code snippet.


key: chroma:document
string_value:
CODE:

class Multimodal1TextRotaryEmbedding(nn.Module):
    def __init__(
        self,
        config: Multimodal1TextConfig,
        device=None,
    ):
        super().__init__()
        self.rope_kwargs = {}
        # BC: "rope_type" was originally "type"
        if hasattr(config, "rope_scaling") and config.rope_scaling is not None:
            self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
        else:
            self.rope_type = "default"
        self.max_seq_len_cached = config.max_position_embeddings
        self.original_max_seq_len = config.max_position_embeddings

        self.config = config
        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]

        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device, **self.rope_kwargs)
        self.register_buffer("inv_freq", inv_freq, persistent=False)
        self.original_inv_freq = self.inv_freq

    def _dynamic_frequency_update(self, position_ids, device):
        """
        dynamic RoPE layers should recompute `inv_freq` in the following situations:
        1 - growing beyond the cached sequence length (allow scaling)
        2 - the current sequence length is in the original scale (avoid losing precision with small sequences)
        """
        seq_len = torch.max(position_ids) + 1
        if seq_len > self.max_seq_len_cached:  # growth
            inv_freq, self.attention_scaling = self.rope_init_fn(
                self.config, device, seq_len=seq_len, **self.rope_kwargs
            )
            self.register_buffer("inv_freq", inv_freq, persistent=False)  # TODO joao: may break with compilation
            self.max_seq_len_cached = seq_len

        if seq_len < self.original_max_seq_len and self.max_seq_len_cached > self.original_max_seq_len:  # reset
            self.register_buffer("inv_freq", self.original_inv_freq, persistent=False)
            self.max_seq_len_cached = self.original_max_seq_len

    @torch.no_grad()
    def forward(self, x, position_ids):
        if "dynamic" in self.rope_type:
            self._dynamic_frequency_update(position_ids, device=x.device)

        # Core RoPE block
        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
        position_ids_expanded = position_ids[:, None, :].float()
        # Force float32 (see https://github.com/huggingface/transformers/pull/29285)
        device_type = x.device.type
        device_type = device_type if isinstance(device_type, str) and device_type != "mps" else "cpu"
        with torch.autocast(device_type=device_type, enabled=False):
            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
            emb = torch.cat((freqs, freqs), dim=-1)
            cos = emb.cos()
            sin = emb.sin()

        # Advanced RoPE types (e.g. yarn) apply a post-processing scaling factor, equivalent to scaling attention
        cos = cos * self.attention_scaling
        sin = sin * self.attention_scaling



SUMMARY:
Here is a concise summary of the provided class code chunk:

**Summary:**
The `Multimodal1TextRotaryEmbedding` class is a PyTorch implementation of a RoPE (Rotary Embedding) layer, which transforms input sequences into dense vector representations. It inherits from PyTorch's `nn.Module`.

**Main Purpose and Functionality:**

* Applies the RoPE transformation to input sequences, utilizing dynamic frequency updates when necessary.
* Can be configured with different rope types, scaling factors, and maximum sequence lengths.

**Key Implementation Details:**

* The `_dynamic_frequency_update` method adjusts the inverse frequency matrix based on sequence length.
* In the `forward` method:
	+ It expands the RoPE frequency table to match the input sequence shape.
	+ Applies the RoPE transformation using the dot product of frequency and position IDs.
	+ Optionally applies post-processing scaling factors (if applicable).

**Notable Dependencies and Requirements:**

* `ROPE_INIT_FUNCTIONS`: a dictionary mapping rope types to initialization functions.
* `Multimodal1TextConfig`: a configuration class for this layer, containing settings such as rope type, scaling factor, and maximum sequence length.


key: chroma:document
string_value:
CODE:

class Multimodal1TextMLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))


SUMMARY:
**Class Summary: Multimodal1TextMLP**

* **Main Purpose and Functionality:** A neural network module for multimodal text processing, utilizing MLP architecture in combination with attention gate.

**Key Implementation Details:**

* MLP (Multi-Layer Perceptron) structure with three linear projections (`gate_proj`, `up_proj`, `down_proj`).
* Gate projection applied to input with attention activation function (`ACT2FN[config.hidden_act]`) followed by up-projection.
* Output from gate and up-projections are combined using a linear transformation.

**Notable Dependencies and Requirements:**

* `nn.Linear`: Linear transformation layers.
* `ACT2FN[config.hidden_act]`: Attention activation function (dependent on hidden act type).
* `config.mlp_bias`: Bias term for linear projections.
* `config.hidden_size` and `config.intermediate_size`: Hidden state and intermediate layer sizes.


key: chroma:document
string_value:
CODE:

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]


SUMMARY:
**Summary:**

Rotates half of the hidden dimensions of input tensor `x` by mirroring the latter half around the first half. The rotated tensor is obtained by concatenating the transformed dimensions in reverse order.

**Implementation Details:**

* Rotates x by splitting it into two halves along the last dimension.
* Inverts and concatenates the second half to create the rotated tensor, preserving the sign of the original values.

**Notable Dependencies/Requirements:**

* Utilizes `torch` library for tensor operations.


key: chroma:document
string_value:
CODE:

def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
    """Applies Rotary Position Embedding to the query and key tensors.

    Args:
        q (`torch.Tensor`): The query tensor.
        k (`torch.Tensor`): The key tensor.
        cos (`torch.Tensor`): The cosine part of the rotary embedding.
        sin (`torch.Tensor`): The sine part of the rotary embedding.
        position_ids (`torch.Tensor`, *optional*):
            Deprecated and unused.
        unsqueeze_dim (`int`, *optional*, defaults to 1):
            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
    Returns:
        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
    """
    cos = cos.unsqueeze(unsqueeze_dim)
    sin = sin.unsqueeze(unsqueeze_dim)
    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)


SUMMARY:
**Rotary Position Embedding Function Summarization**

The `apply_rotary_pos_emb` function applies Rotary Position Embedding to query and key tensors. It takes in two input tensors `q` and `k`, along with cosine and sine parts of the rotary embedding, `cos` and `sin`.

**Summary:**

* Applies Rotary Position Embedding to query (`q`) and key (`k`) tensors
* Incorporates two separate channels (cosine and sine) into the embedding process

**Notable Implementation Details:**

* Rotating half the input tensor using `rotate_half(q)` or `rotate_half(k)`
* Broadcasting the cosine and sine parts along an additional dimension (unsqueeze_dim)
* Composing the embedding process with the query and key tensors

**Dependencies:**

* `torch.Tensor`: Query, Key, Cosine, Sine tensors
* `rotate_half`: A function not shown in this code snippet (implies a custom implementation)

The function returns a tuple containing the rotary-embbedded query tensor (`q_embed`) and key tensor (`k_embed`).


key: chroma:document
string_value:
CODE:

def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
    """
    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
    """
    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
    if n_rep == 1:
        return hidden_states
    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)


SUMMARY:
**Summary**

The `repeat_kv` function transforms a 4D `torch.Tensor` of shape `(batch, num_key_value_heads, seqlen, head_dim)` to match the structure of `torch.repeat_interleave(x, dim=1, repeats=n_rep)`, essentially interleaving key-value pairs.

**Key Implementation Details**

* Repeats each row (`num_key_value_heads`) `n_rep` times along the new axis (dimension 3).
* New shape: `(batch, num_attention_heads * n_rep, seqlen, head_dim)` where `num_attention_heads` is equivalent to `num_key_value_heads`.
* Implemented using tensor expansion and reshaping.

**Notable Dependencies and Requirements**

* `torch.Tensor`
* No external dependencies referenced
* Adheres to PyTorch's `repeat_interleave` API pattern.


key: chroma:document
string_value:
CODE:

def eager_attention_forward(
    module: nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    scaling: float,
    dropout: float = 0.0,
    **kwargs,
):
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)

    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
    if attention_mask is not None:
        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
        attn_weights = attn_weights + causal_mask

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
    attn_output = torch.matmul(attn_weights, value_states)
    attn_output = attn_output.transpose(1, 2).contiguous()



SUMMARY:
**Function Summary: Eager Attention Forward**

*Main Purpose and Functionality:*
The `eager_attention_forward` function implements the forward pass of a multi-head attention mechanism using linear transformations and softmax scaling. It takes in query, key, value tensors, and optional attention mask, and returns the output and corresponding attention weights.

**Key Implementation Details:**

* Uses `torch.matmul` to compute attention weights and output
* Applies softmax scaling and normalizes attention weights
* Includes causal self-attention (if attention mask is provided)
* Dropout layer for regularization

**Dependencies and Requirements:**

* `nn.Module`: The input module
* `torch.Tensor`: Query, key, value tensors
* `Optional[torch.Tensor]`: Attention mask tensor
* `kwargs`: Additional keyword arguments (not used in this implementation)

**Method Signatures and Interfaces:**

* `torch.nn.functional.softmax()`
* `torch.nn.functional.dropout()`
	+ `p`: dropout probability
	+ `training`: training mode


key: chroma:document
string_value:
CODE:

class Multimodal1TextAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(self, config: Multimodal1TextConfig, layer_idx: int):
        super().__init__()
        self.config = config
        self.layer_idx = layer_idx
        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
        self.scaling = self.head_dim**-0.5
        self.attention_dropout = config.attention_dropout
        self.is_causal = True

        self.q_proj = nn.Linear(
            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
        )
        self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
        self.v_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
        self.o_proj = nn.Linear(
            config.num_attention_heads * self.head_dim, config.hidden_size, bias=config.attention_bias
        )

    def forward(
        self,
        hidden_states: torch.Tensor,
        position_embeddings: Tuple[torch.Tensor, torch.Tensor],
        attention_mask: Optional[torch.Tensor],
        past_key_value: Optional[Cache] = None,
        cache_position: Optional[torch.LongTensor] = None,
        **kwargs: Unpack[FlashAttentionKwargs],
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
        input_shape = hidden_states.shape[:-1]
        hidden_shape = (*input_shape, -1, self.head_dim)

        query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)

        cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)

        if past_key_value is not None:
            # sin and cos are specific to RoPE models; cache_position needed for the static cache
            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)

        attention_interface: Callable = eager_attention_forward
        if self.config._attn_implementation != "eager":
            if self.config._attn_implementation == "sdpa" and kwargs.get("output_attentions", False):
                logger.warning_once(
                    "`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to "
                    'eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.'
                )
            else:
                attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]

        attn_output, attn_weights = attention_interface(
            self,
            query_states,
            key_states,
            value_states,
            attention_mask,
            dropout=0.0 if not self.training else self.attention_dropout,
            scaling=self.scaling,
            **kwargs,
        )

        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)


SUMMARY:
**Multimodal 1 Text Attention Class Summary**

### Purpose and Functionality
The `Multimodal1TextAttention` class implements multi-headed attention from the 'Attention Is All You Need' paper. It's a PyTorch module that takes in input data, position embeddings, and attention mask to compute attention weights and output.

### Key Implementation Details

* Three linear projections: `q_proj`, `k_proj`, and `v_proj` for queries, keys, and values respectively.
* Two more linear projections: `o_proj` for output and `attention_dropout` for regularization.

### Notable Dependencies and Requirements
* Requires `Multimodal1TextConfig` configuration for hyperparameters.
* Inherents from `nn.Module` architecture.
* Depends on `eager_attention_forward` or `ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]` attention interfaces.


key: chroma:document
string_value:
CODE:

class Multimodal1TextDecoderLayer(nn.Module):
    def __init__(self, config: Multimodal1TextConfig, layer_idx: int):
        super().__init__()
        self.hidden_size = config.hidden_size

        self.self_attn = Multimodal1TextAttention(config=config, layer_idx=layer_idx)

        self.mlp = Multimodal1TextMLP(config)
        self.input_layernorm = Multimodal1TextRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
        self.post_attention_layernorm = Multimodal1TextRMSNorm(config.hidden_size, eps=config.rms_norm_eps)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_value: Optional[Cache] = None,
        output_attentions: Optional[bool] = False,
        use_cache: Optional[bool] = False,
        cache_position: Optional[torch.LongTensor] = None,
        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
        **kwargs: Unpack[FlashAttentionKwargs],
    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            cache_position=cache_position,
            position_embeddings=position_embeddings,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

        outputs = (hidden_states,)
        if output_attentions:
            outputs += (self_attn_weights,)



SUMMARY:
**Multimodal1TextDecoderLayer**

**Main Purpose and Functionality**
The `Multimodal1TextDecoderLayer` is a neural network layer that decodes multimodal text inputs. It implements self-attention mechanism followed by fully connected (MLP) layer.

**Key Implementation Details**

* Self-Attention Mechanism: `self_attn`
	+ Input: `hidden_states`, `attention_mask`, `position_ids`, `past_key_value`, ... ( keyword arguments )
	+ Output: `(hidden_states, self_attn_weights)`
* Fully Connected Layer: `mlp`
	+ Input: `hidden_states`
	+ Output: `hidden_states`

**Notable Dependencies and Requirements**

* `Multimodal1TextConfig`: configuration object for the layer
* `Multimodal1TextAttention`: attention mechanism class
* `Multimodal1TextRMSNorm` and `Multimodal1TextMLP`: normalization and fully connected layer classes respectively
* `torch` library and its tensors for computations


key: chroma:document
string_value:
CODE:

class SuperRMSNorm(nn.Module):
    def __init__(self, hidden_size, eps=1e-6):
        """
        SuperRMSNorm is equivalent to T5LayerNorm
        """
        super().__init__()
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.variance_epsilon = eps

    def forward(self, hidden_states):
        input_dtype = hidden_states.dtype
        hidden_states = hidden_states.to(torch.float32)
        variance = hidden_states.pow(2).mean(-1, keepdim=True)
        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
        return self.weight * hidden_states.to(input_dtype)

    def extra_repr(self):


SUMMARY:
**Concise Summary:**

The `SuperRMSNorm` class is an equivalent implementation of T5LayerNorm. It normalizes the input tensor using RMS Normalization, which scales and shifts the input to have zero mean and unit variance.

* Primary purpose:
	+ Normalizes input tensor using RMS Normalization
	+ Returns scaled and shifted output
* Key implementation details:
	+ Computes variance of input using `torch.rsqrt`
	+ Applies scaling and shifting factors using `self.weight` and `self.variance_epsilon`
* Notable dependencies or requirements:
	+ Uses PyTorch's nn.Module and nn.Parameter for computation
	+ Requires `hidden_size` parameter for weight initialization
	+ Inherited the `__init__` method from PyTorch's `nn.Module`

**Key Interface:**

* `forward(hidden_states)`: Normalizes input tensor using RMS Normalization
* `extra_repr()`: Returns string representation of the module's weights and epsilon value

Note that external dependencies are not listed as they are implicit, being provided by PyTorch.


key: chroma:document
string_value:
CODE:

class SuperRotaryEmbedding(nn.Module):
    def __init__(
        self,
        config: SuperConfig,
        device=None,
    ):
        super().__init__()
        self.rope_kwargs = {}
        # BC: "rope_type" was originally "type"
        if hasattr(config, "rope_scaling") and config.rope_scaling is not None:
            self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
        else:
            self.rope_type = "default"
        self.max_seq_len_cached = config.max_position_embeddings
        self.original_max_seq_len = config.max_position_embeddings

        self.config = config
        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]

        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device, **self.rope_kwargs)
        self.register_buffer("inv_freq", inv_freq, persistent=False)
        self.original_inv_freq = self.inv_freq

    def _dynamic_frequency_update(self, position_ids, device):
        """
        dynamic RoPE layers should recompute `inv_freq` in the following situations:
        1 - growing beyond the cached sequence length (allow scaling)
        2 - the current sequence length is in the original scale (avoid losing precision with small sequences)
        """
        seq_len = torch.max(position_ids) + 1
        if seq_len > self.max_seq_len_cached:  # growth
            inv_freq, self.attention_scaling = self.rope_init_fn(
                self.config, device, seq_len=seq_len, **self.rope_kwargs
            )
            self.register_buffer("inv_freq", inv_freq, persistent=False)  # TODO joao: may break with compilation
            self.max_seq_len_cached = seq_len

        if seq_len < self.original_max_seq_len and self.max_seq_len_cached > self.original_max_seq_len:  # reset
            self.register_buffer("inv_freq", self.original_inv_freq, persistent=False)
            self.max_seq_len_cached = self.original_max_seq_len

    @torch.no_grad()
    def forward(self, x, position_ids):
        if "dynamic" in self.rope_type:
            self._dynamic_frequency_update(position_ids, device=x.device)

        # Core RoPE block
        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
        position_ids_expanded = position_ids[:, None, :].float()
        # Force float32 (see https://github.com/huggingface/transformers/pull/29285)
        device_type = x.device.type
        device_type = device_type if isinstance(device_type, str) and device_type != "mps" else "cpu"
        with torch.autocast(device_type=device_type, enabled=False):
            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
            emb = torch.cat((freqs, freqs), dim=-1)
            cos = emb.cos()
            sin = emb.sin()

        # Advanced RoPE types (e.g. yarn) apply a post-processing scaling factor, equivalent to scaling attention
        cos = cos * self.attention_scaling
        sin = sin * self.attention_scaling



SUMMARY:
Here is a concise summary of the provided code chunk:

**Class**: `SuperRotaryEmbedding` (a PyTorch module)

**Purpose**: The `SuperRotaryEmbedding` class implements a self-attention mechanism using the RoPE (Rotary Position Embedding) method, which allows for efficient and scalable attention over long sequences.

**Key Features**:

* Supports dynamic frequency updates based on sequence length.
* Can be configured with "dynamic" scaling type.
* Includes post-processing scaling factor for advanced types like Yarn.
* Applies cosine and sine functions to RoPE embeddings.

**Notable Attributes and Methods**:

* `rope_kwargs`: configuration dictionary for RoPE initialization.
* `max_seq_len_cached` and `original_max_seq_len`: cached sequence length variables.
* `_dynamic_frequency_update`: method updating frequency for dynamic scaling.
* `forward`: the primary forward pass method taking input `x` and `position_ids`.

**Dependencies**: 
- PyTorch ( TorchAutocast )
- Transformers (ROPE_INIT_FUNCTIONS)


key: chroma:document
string_value:
CODE:

class SuperMLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))


SUMMARY:
**Summary:**

The `SuperMLP` class is a PyTorch implementation of a multi-layer perceptron (MLP) network, specifically designed for the Super Glance model. It inherits from `nn.Module`.

**Key Functionality:**

- Composes two projection matrices (`gate_proj` and `up_proj`) using the `act_fn` activation function.
- Applies these projections sequentially to the input `x` in a bottom-up manner.

**Implementation Details:**

- `forward` function uses `down_proj`, `gate_proj`, `up_proj`, and `act_fn` to process inputs.
 - Key modules:
  - `self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)`
  - `self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)` 
  - `down_proj = nn.Linear=self.intermediate_size, self.hidden_size, bias=config.mlp_bias)`

**Notable Dependencies/Requirements:**

- `config`: an object containing hyperparameters and configuration settings.
- `ACT2FN`: a dictionary mapping activation functions to their respective PyTorch implementations.


key: chroma:document
string_value:
CODE:

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]


SUMMARY:
**Summary:**

Rotates the last half of hidden dimensions of input tensor `x` by reflecting it through the middle dimension and concatenating with its original first half.

**Key Implementation Details:**

- Rotates tensor using slicing (`x[..., : x.shape[-1] // 2]`)
- Reflects half of the tensor across the midpoint using `-x2`
- Concatenates rotated and original halves along the last axis (-1)
- Uses PyTorch's `torch.cat` function for concatenation

**Notable Dependencies:**

- **Input:** `x`: expected input tensor with shape that allows slicing
- **Output:** Rotated tensor (shape dependent on input)

Note: This summary focuses on the key functionality and implementation details of the provided code chunk. It aims to be concise while preserving essential information for understanding the function's purpose and behavior.


key: chroma:document
string_value:
CODE:

def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
    """Applies Rotary Position Embedding to the query and key tensors.

    Args:
        q (`torch.Tensor`): The query tensor.
        k (`torch.Tensor`): The key tensor.
        cos (`torch.Tensor`): The cosine part of the rotary embedding.
        sin (`torch.Tensor`): The sine part of the rotary embedding.
        position_ids (`torch.Tensor`, *optional*):
            Deprecated and unused.
        unsqueeze_dim (`int`, *optional*, defaults to 1):
            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
    Returns:
        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
    """
    cos = cos.unsqueeze(unsqueeze_dim)
    sin = sin.unsqueeze(unsqueeze_dim)
    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)


SUMMARY:
**Summary: RotateQueryEmbedding**

Applies Rotary Position Embedbing to query and key tensors.

**Functionality:**

1. Applies Rotary Position Embedding to query (`q`) and key (`k`) tensors:
	* Scales `cos` and `sin` by the specified unsqueeze dimension using `unsqueeze`.
	* Computes rotated embeddings as `(q * cos) + (rotate_half(q) * sin)` for `q_embed`, and similarly for `k_embed`.
2. Returns a tuple containing the rotated query embedding (`q_embed`) and key embedding (`k_embed`).

**Implementation Details:**

* Uses `rotate_half` function to compute half rotation of input tensors.
* Applies unsqueeze dimension to broadcast tensors with specific shapes.

**Notable Dependencies/Requirements:**

* Input tensors `q`, `k`, `cos`, and `sin` must be of type `torch.Tensor`.
* The `rotate_half` function is assumed to be defined elsewhere in the codebase.

**Return Value:** A tuple containing the rotated query embedding (`q_embed`) and key embedding (`k_embed`).


key: chroma:document
string_value:
CODE:

def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
    """
    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
    """
    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
    if n_rep == 1:
        return hidden_states
    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)


SUMMARY:
**Summary**

`repeat_kv` function:
Replicates hidden states in the multi-head attention mechanism.

**Main Purpose and Functionality**
Transposes the input `hidden_states` from `(batch, num_key_value_heads, seqlen, head_dim)` to `(batch, num_attention_heads, seqlen, head_dim)`

while maintaining the same batch dimensionality. When `n_rep == 1`, original tensor is returned.

**Implementation Details**

* Replicates hidden states in the sequence dimension using `expand`
* Reshapes the tensor to (batch, num_key_value_heads * n_rep, seqlen, head_dim)

**Notable Dependencies and Requirements**
Replaces `torch.repeat_interleave` functionality with explicit expansion and reshaping.

Note: External dependency is no additional external library besides PyTorch.


key: chroma:document
string_value:
CODE:

def eager_attention_forward(
    module: nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    scaling: float,
    dropout: float = 0.0,
    **kwargs,
):
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)

    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
    if attention_mask is not None:
        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
        attn_weights = attn_weights + causal_mask

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
    attn_output = torch.matmul(attn_weights, value_states)
    attn_output = attn_output.transpose(1, 2).contiguous()



SUMMARY:
**Concise Summary:**

Eager Attention Forward Function
=====================================

### Purpose and Functionality

The `eager_attention_forward` function computes the attention output for a given query, key, and value tensor in an eager attention mechanism. It applies masked self-attention to account for causal dependencies.

### Key Implementation Details

* Computes attention weights using QK^T (query-key dot product) and softmax normalization
* Masks out causal dependencies using `attention_mask`
* Applies dropout regularization
* Outputs attended value tensor using transformed weights

### Notable Dependencies or Requirements

* `nn.Module`: input module object
* `torch.Tensor`: query, key, value tensors
* `attention_mask`: optional attention mask tensor
* `scaling`: scaling factor for weights
* `dropout`: dropout rate (default: 0.0)
* `kwargs`: additional keyword arguments (not used)

### Interface

`eager_attention_forward(output, weights)`

where `output` is the attended value tensor and `weights` is a tensor of attention weights.

### External Dependencies

None


key: chroma:document
string_value:
CODE:

class SuperAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(self, config: SuperConfig, layer_idx: int):
        super().__init__()
        self.config = config
        self.layer_idx = layer_idx
        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
        self.scaling = self.head_dim**-0.5
        self.attention_dropout = config.attention_dropout
        self.is_causal = True

        self.q_proj = nn.Linear(
            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
        )
        self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
        self.v_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
        self.o_proj = nn.Linear(
            config.num_attention_heads * self.head_dim, config.hidden_size, bias=config.attention_bias
        )

    def forward(
        self,
        hidden_states: torch.Tensor,
        position_embeddings: Tuple[torch.Tensor, torch.Tensor],
        attention_mask: Optional[torch.Tensor],
        past_key_value: Optional[Cache] = None,
        cache_position: Optional[torch.LongTensor] = None,
        **kwargs: Unpack[FlashAttentionKwargs],
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
        input_shape = hidden_states.shape[:-1]
        hidden_shape = (*input_shape, -1, self.head_dim)

        query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)

        cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)

        if past_key_value is not None:
            # sin and cos are specific to RoPE models; cache_position needed for the static cache
            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)

        attention_interface: Callable = eager_attention_forward
        if self.config._attn_implementation != "eager":
            if self.config._attn_implementation == "sdpa" and kwargs.get("output_attentions", False):
                logger.warning_once(
                    "`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to "
                    'eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.'
                )
            else:
                attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]

        attn_output, attn_weights = attention_interface(
            self,
            query_states,
            key_states,
            value_states,
            attention_mask,
            dropout=0.0 if not self.training else self.attention_dropout,
            scaling=self.scaling,
            **kwargs,
        )

        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)


SUMMARY:
**SuperAttention Class Summary**

The `SuperAttention` class implements multi-headed attention mechanism based on the "Attention Is All You Need" paper. It is a PyTorch module that extends the traditional self-attention mechanism.

### Key Implementation Details:

1. **Projector Layers**: The class defines four projector layers:
   - `q_proj`: query projection
   - `k_proj`: key projection
   - `v_proj`: value projection
   - `o_proj`: output projection

2. **Attention Dropout and Scaling**: The attention mechanism uses a scaling factor computed as `head_dim**-0.5`, applied after computing the attention.

3. **Eager Attention vs Scaled Dot-Product Attention (SDPA)**: The class distinguishes between two implementations:
   - Eager attention
   - SDPA, which is backed by an implementation dependent on the `_attn_implementation` attribute

### Notable Dependencies/Requirements:

1. `torch.nn.Linear`: used to define linear projector layers.
2. `super().__init__`: inherited from PyTorch's neural network module (`nn.Module`).
3. External dependencies:
	* `torch`: necessary for tensor computations and module initialization.

### Interfaces:
The class uses the following interfaces:
- `eager_attention_forward`
- `ALL ATTENTION_FUNCTIONS` dictionary, backed by an implementation dependent on the `_attn_implementation` attribute.


key: chroma:document
string_value:
CODE:

class SuperDecoderLayer(nn.Module):
    def __init__(self, config: SuperConfig, layer_idx: int):
        super().__init__()
        self.hidden_size = config.hidden_size

        self.self_attn = SuperAttention(config=config, layer_idx=layer_idx)

        self.mlp = SuperMLP(config)
        self.input_layernorm = SuperRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
        self.post_attention_layernorm = SuperRMSNorm(config.hidden_size, eps=config.rms_norm_eps)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_value: Optional[Cache] = None,
        output_attentions: Optional[bool] = False,
        use_cache: Optional[bool] = False,
        cache_position: Optional[torch.LongTensor] = None,
        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
        **kwargs: Unpack[FlashAttentionKwargs],
    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            cache_position=cache_position,
            position_embeddings=position_embeddings,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

        outputs = (hidden_states,)
        if output_attentions:
            outputs += (self_attn_weights,)



SUMMARY:
**Summary:**

`SuperDecoderLayer` is a PyTorch implementation of the decoder layer in a transformer model. It inherits from `nn.Module`.

**Primary Purpose:** This class handles the decoding process of a sequence-to-sequence model, consisting of self-attention and fully connected (MLP) components.

**Key Implementation Details:**

* Self-attention mechanism using `SuperAttention` with custom settings.
* Post-attention layer normalization (`SuperRMSNorm`) applied to intermediate results.
* Fully connected layers using `SuperMLP` with the same configuration.
* Input and post-attention layer normalizations applying to hidden states.

**Notable Dependencies:**

* `SuperConfig`: defines configuration parameters, including hidden size and RMS norm epsilon.
* `SuperAttention`, `SuperRMSNorm`, `SuperMLP`: custom modules implementing specific attention mechanisms, normalization techniques, and fully connected layers.

This code is part of a larger sequence-to-sequence transformer model, likely for machine translation tasks.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_my_new_model2.py


key: chroma:document
string_value:
CODE:

class MyNewModel2RMSNorm(nn.Module):
    def __init__(self, dim: int, eps: float = 1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.zeros(dim))

    def _norm(self, x):
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

    def forward(self, x):
        output = self._norm(x.float())
        # Llama does x.to(float16) * w whilst MyNewModel2 is (x * w).to(float16)
        # See https://github.com/huggingface/transformers/pull/29402
        output = output * (1.0 + self.weight.float())
        return output.type_as(x)

    def extra_repr(self):


SUMMARY:
**Summary: MyNewModel2RMSNorm Class**

The `MyNewModel2RMSNorm` class is a PyTorch neural network module that implements the RMS Norm normalization technique. It normalizes input tensors using their mean squared values, scaling by a learned weight.

**Key Features:**

* **Primary Purpose:** Normalize input tensors using their mean squared values.
* **Implementation Details:** The `_norm` method computes the normalized output by multiplying the input `x` with the square root of its mean magnitude plus epsilon. The `forward` method applies this normalization and then scales the result by a learned weight vector.
* **Notable Dependencies:**
	+ `nn.Module`: PyTorch neural network module base class.
	+ `torch.nn.Parameter`: Learnable parameter container.
	+ `torch`: Torch library for PyTorch functions (e.g., `rsqrt`, `pow`).
* **Key Method Signatures:**
	+ `_norm(x)`: Compute normalized output by x.
	+ `forward(x)`: Apply RMS Norm normalization and scaling.
	+ `extra_repr()`: Return model weight shape and epsilon value as string representation.

**Dependencies:**

* `torch`
* External dependencies not applicable (self-contained PyTorch implementation).


key: chroma:document
string_value:
CODE:

class MyNewModel2MLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))


SUMMARY:
**Summary:**

The `MyNewModel2MLP` class is an implementation of a multi-layer perceptron (MLP) using PyTorch's `nn.Module`. The model takes in input data `x`, applies an activation function (`self.act_fn`) to the linear transformation result of two projection layers (`self.gate_proj` and `self.up_proj`), and then passes the output through another projection layer (`self.down_proj`). The primary purpose is to transform the input data.

**Key Implementation Details:**

* `ACT2FN[config.hidden_act]`: uses a configuration-dependent activation function
* Compositions of linear transformations and element-wise multiplications with activation functions

**Notable Dependencies or Requirements:**

* `nn.Linear`
* `nn.Module`
* `nn.init` (activation functions)
* PyTorch's dynamic computation graph

**Main Purpose and Functionality:**
The model transforms input data using a composition of linear projections, followed by an element-wise multiplication with an activation function applied to the gate projection result.


key: chroma:document
string_value:
CODE:

class MyNewModel2RotaryEmbedding(nn.Module):
    def __init__(
        self,
        config: MyNewModel2Config,
        device=None,
    ):
        super().__init__()
        self.rope_kwargs = {}
        # BC: "rope_type" was originally "type"
        if hasattr(config, "rope_scaling") and config.rope_scaling is not None:
            self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
        else:
            self.rope_type = "default"
        self.max_seq_len_cached = config.max_position_embeddings
        self.original_max_seq_len = config.max_position_embeddings

        self.config = config
        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]

        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device, **self.rope_kwargs)
        self.register_buffer("inv_freq", inv_freq, persistent=False)
        self.original_inv_freq = self.inv_freq

    def _dynamic_frequency_update(self, position_ids, device):
        """
        dynamic RoPE layers should recompute `inv_freq` in the following situations:
        1 - growing beyond the cached sequence length (allow scaling)
        2 - the current sequence length is in the original scale (avoid losing precision with small sequences)
        """
        seq_len = torch.max(position_ids) + 1
        if seq_len > self.max_seq_len_cached:  # growth
            inv_freq, self.attention_scaling = self.rope_init_fn(
                self.config, device, seq_len=seq_len, **self.rope_kwargs
            )
            self.register_buffer("inv_freq", inv_freq, persistent=False)  # TODO joao: may break with compilation
            self.max_seq_len_cached = seq_len

        if seq_len < self.original_max_seq_len and self.max_seq_len_cached > self.original_max_seq_len:  # reset
            self.register_buffer("inv_freq", self.original_inv_freq, persistent=False)
            self.max_seq_len_cached = self.original_max_seq_len

    @torch.no_grad()
    def forward(self, x, position_ids):
        if "dynamic" in self.rope_type:
            self._dynamic_frequency_update(position_ids, device=x.device)

        # Core RoPE block
        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
        position_ids_expanded = position_ids[:, None, :].float()
        # Force float32 (see https://github.com/huggingface/transformers/pull/29285)
        device_type = x.device.type
        device_type = device_type if isinstance(device_type, str) and device_type != "mps" else "cpu"
        with torch.autocast(device_type=device_type, enabled=False):
            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
            emb = torch.cat((freqs, freqs), dim=-1)
            cos = emb.cos()
            sin = emb.sin()

        # Advanced RoPE types (e.g. yarn) apply a post-processing scaling factor, equivalent to scaling attention
        cos = cos * self.attention_scaling
        sin = sin * self.attention_scaling



SUMMARY:
**Summary:**

This is a PyTorch implementation of the RoPE (Rotated Polynomial Embeddings) model, which is an attention mechanism. The `MyNewModel2RotaryEmbedding` class encapsulates the model's functionality.

**Main Purpose and Functionality:**
The primary purpose is to compute the attention weights for a given input sequence using the RoPE algorithm.

**Key Implementation Details:**

1.  **Initialization**: The model takes in a configuration object `config` and an optional device (default=None). It initializes several internal variables, including `rope_type`, `max_seq_len_cached`, and `original_max_seq_len`.
2.  **Dynamic Frequency Update**: The `_dynamic_frequency_update` method adjusts the `inv_freq` tensor based on the sequence length to maintain precision.
3.  **Forward Pass**: The `forward` method computes the attention weights using the RoPE algorithm. It performs the following steps:
    *   Updates the frequency tensor if necessary.
    *   Computes the cosine and sine components of the embeddings.
    *   Applies post-processing scaling factors to the cosine and sine components.

**Notable Dependencies or Requirements:**

1.  `ROPE_INIT_FUNCTIONS`: A dictionary mapping `rope_type` values to their corresponding initialization functions.
2.  `MyNewModel2Config`: The configuration class for this model, containing settings like `rope_scaling`.
3.


key: chroma:document
string_value:
CODE:

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]


SUMMARY:
**Summary**

Rotates half of the hidden dimensions of input tensor `x` by mirroring the latter half around the middle dimension.

**Key Details**

* Rotates half dimensions using slicing (`x[..., : x.shape[-1] // 2]`, `x[..., x.shape[-1] // 2 :]`)
* Mirrors half with negation (`-x2`) followed by concatenation along last axis (-1)
* Reforms tensor shape with resulting rotation applied

**Related Dependencies**

* torch for tensor operations and cat function implementation
* no external dependencies noted


key: chroma:document
string_value:
CODE:

def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
    """Applies Rotary Position Embedding to the query and key tensors.

    Args:
        q (`torch.Tensor`): The query tensor.
        k (`torch.Tensor`): The key tensor.
        cos (`torch.Tensor`): The cosine part of the rotary embedding.
        sin (`torch.Tensor`): The sine part of the rotary embedding.
        position_ids (`torch.Tensor`, *optional*):
            Deprecated and unused.
        unsqueeze_dim (`int`, *optional*, defaults to 1):
            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
    Returns:
        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
    """
    cos = cos.unsqueeze(unsqueeze_dim)
    sin = sin.unsqueeze(unsqueeze_dim)
    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)


SUMMARY:
**Summary: Rotary Position Embedding Function**

Applies Rotary Position Embedding to query and key tensors.

### Main Purpose and Functionality
Applys Rotary Position Embedding to `q` and `k` tensors using `(q * cos) + (rotate_half(q) * sin)` for `q_embed` and `(k * cos) + (rotate_half(k) * sin)` for `k_embed`.

### Key Implementation Details

* Cosine (`cos`) and sine (`sin`) tensors are unsqueezed along specified dimension.
* Use of `rotate_half()` function.
* Rotary Position Embedding formula applied: `(q * cos) + (rotate_half(q) * sin)`.

### Notable Dependencies or Requirements

* `torch.Tensor`
* `rotate_half()` function, assuming it's defined elsewhere in the codebase.
* No explicit external dependencies listed.

**Method Signatures and Interfaces**

* None explicitly listed, but implies usage of `torch.Tensor` interface.


key: chroma:document
string_value:
CODE:

def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
    """
    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
    """
    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
    if n_rep == 1:
        return hidden_states
    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)


SUMMARY:
**Summary**

Repeats key-value pairs in the `hidden_states` tensor along the attention heads dimension.

**Functionality**

- Takes in:
  - `torch.Tensor hidden_states`: Input tensor with shape (batch, num_key_value_heads, seqlen, head_dim)
  - `int n_rep`: Number of times to repeat each key-value pair
- Returns a new tensor with shape (batch, num_attention_heads, seqlen, head_dim)

**Implementation Details**

- Repeats `hidden_states` by expanding along the fourth dimension and repeating that repeat `n_rep` times.

**Dependencies and Requirements**

- Uses PyTorch's tensor operations and does not require any external dependencies.


key: chroma:document
string_value:
CODE:

def eager_attention_forward(
    module: nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    scaling: float,
    dropout: float = 0.0,
    **kwargs,
):
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)

    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
    if attention_mask is not None:
        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
        attn_weights = attn_weights + causal_mask

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
    attn_output = torch.matmul(attn_weights, value_states)
    attn_output = attn_output.transpose(1, 2).contiguous()



SUMMARY:
**Concise Summary**

### Function Purpose and Functionality
The `eager_attention_forward` function implements the Eager Attention mechanism, which generates weighted sums of key-value pairs in a neural network module (`module`).

Primary purpose: Compute attention weights and output.

### Key Implementation Details

*   Computes attention weights using matrix multiplication with scaling factor (`scaling`) applied.
*   Applies dropout to attention weights.
*   Merges attention weights with causal mask (if provided) for masked token handling.
*   Softmax-activation of attention weights.
*   computes attention output by multiplying attention weights with value states.

### Notable Dependencies and Requirements

*   `torch` tensor modules (`nn.Module`, `torch.Tensor`)
*   `torch.functional` module (`nn.functional.softmax`, `nn.functional.dropout`)
*   External dependency: `repeat_kv` function (not provided in the code snippet)
*   Implemented interface: `module.num_key_value_groups`
*   Optional dependencies:
    *   Relies on causal mask for masked token handling. If not available, no additional operations are performed.
        |


key: chroma:document
string_value:
CODE:

class MyNewModel2Attention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(self, config: MyNewModel2Config, layer_idx: int):
        super().__init__()
        self.config = config
        self.layer_idx = layer_idx
        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
        self.scaling = self.head_dim**-0.5
        self.attention_dropout = config.attention_dropout
        self.is_causal = True

        self.q_proj = nn.Linear(
            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
        )
        self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
        self.v_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
        self.o_proj = nn.Linear(
            config.num_attention_heads * self.head_dim, config.hidden_size, bias=config.attention_bias
        )

    def forward(
        self,
        hidden_states: torch.Tensor,
        position_embeddings: Tuple[torch.Tensor, torch.Tensor],
        attention_mask: Optional[torch.Tensor],
        past_key_value: Optional[Cache] = None,
        cache_position: Optional[torch.LongTensor] = None,
        **kwargs: Unpack[FlashAttentionKwargs],
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
        input_shape = hidden_states.shape[:-1]
        hidden_shape = (*input_shape, -1, self.head_dim)

        query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)

        cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)

        if past_key_value is not None:
            # sin and cos are specific to RoPE models; cache_position needed for the static cache
            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)

        attention_interface: Callable = eager_attention_forward
        if self.config._attn_implementation != "eager":
            if self.config._attn_implementation == "sdpa" and kwargs.get("output_attentions", False):
                logger.warning_once(
                    "`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to "
                    'eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.'
                )
            else:
                attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]

        attn_output, attn_weights = attention_interface(
            self,
            query_states,
            key_states,
            value_states,
            attention_mask,
            dropout=0.0 if not self.training else self.attention_dropout,
            scaling=self.scaling,
            **kwargs,
        )

        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)


SUMMARY:
**Summary:**

The `MyNewModel2Attention` class is an implementation of multi-headed attention from the 'Attention Is All You Need' paper. This class takes in configuration parameters and layer index and consists of three projection layers (`q_proj`, `k_proj`, `v_proj`) for queries, keys, and values respectively.

**Key Implementation Details:**

*   Uses scaled-dot product attention with linear transformations and position embeddings.
*   Supports both eager and static cache implementations for past key-value updates.
*   Scales the input to match the attention head dimension via self-scaling.

**Notable Dependencies or Requirements:**

*   `torch` with its tensor operations and `nn.Linear`.
*   `AllAttentionFunctions` map with implemented scaling dot product attention functions (e.g., `ScaledDotProductAttention`, `MatMulAttention`).
*   `MyNewModel2Config` with necessary configuration parameters.


key: chroma:document
string_value:
CODE:

class MyNewModel2DecoderLayer(nn.Module):
    def __init__(self, config: MyNewModel2Config, layer_idx: int):
        super().__init__()
        self.hidden_size = config.hidden_size

        self.self_attn = MyNewModel2Attention(config=config, layer_idx=layer_idx)

        self.mlp = MyNewModel2MLP(config)
        self.input_layernorm = MyNewModel2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
        self.post_attention_layernorm = MyNewModel2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_value: Optional[Cache] = None,
        output_attentions: Optional[bool] = False,
        use_cache: Optional[bool] = False,
        cache_position: Optional[torch.LongTensor] = None,
        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
        **kwargs: Unpack[FlashAttentionKwargs],
    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            cache_position=cache_position,
            position_embeddings=position_embeddings,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

        outputs = (hidden_states,)
        if output_attentions:
            outputs += (self_attn_weights,)



SUMMARY:
**MyNewModel2DecoderLayer Class Summary**

* **Primary Purpose**: A PyTorch module implementing the decoder layer of a transformer model, handling self-attention and feed-forward networks.

* **Key Implementation Details**:
	+ `self_attn` : MyNewModel2Attention module for self-attention mechanism.
	+ `mlp` : MyNewModel2MLP module for fully connected feed-forward network.
	+ `input_layernorm` and `post_attention_layernorm`: Two RMSNorm layers for layer normalization.

* **Notable Dependencies/Requirements**:
	+ `MyNewModel2Config` class, providing configuration parameters.
	+ `MyNewModel2Attention`, `MyNewModel2MPM`, and `RMSNorm` classes from surrounding codebase.

**Method Signatures:**

- `forward`: Takes in input `hidden_states`, `attention_mask`, `position_ids`, and optional keyword arguments, returning a tuple of `(output, attention_weights)`.

**External Dependencies:**


key: chroma:document
string_value:
CODE:

class Multimodal2VisionAttention(CLIPAttention):


SUMMARY:
**Summary:**

This `Multimodal2VisionAttention` class is a subclass of `CLIPAttention`, implying it is implementing the Attention mechanism specifically designed for multimodal 2 (image and text) input in the context of CLIP (Visual-BERT) models.

**Key Features:**

* Subclass of `CLIPAttention`
* Primary purpose: Implementing multimodal 2 attention for image-text pairs

**Implementation Details:**

* No explicit key implementation details are provided, suggesting that this class might be a placeholder or an abstract implementation that needs to be completed by its subclasses.

**Notable Dependencies and Requirements:**

* Inherited/from `CLIPAttention`: Attention mechanism, likely containing methods like `compute_score()` and `compute вним()`.
* External Dependencies:
	+ None explicitly mentioned, but it is assumed to rely on the CLIP library or similar models for multimodal processing.


key: docstring
string_value:
# Check that adding the second base class correctly set the parent, even though in clip it does not have the "Vision" part


key: chroma:document
string_value:
CODE:
# Check that adding the second base class correctly set the parent, even though in clip it does not have the "Vision" part
class Multimodal2VisionSdpaAttention(CLIPSdpaAttention, Multimodal2VisionAttention):


SUMMARY:
**Class Summary**

`Multimodal2VisionSdpaAttention`

 inherites from `CLIPSdpaAttention` and `Multimodal2VisionAttention`. This class appears to implement a multimodal attention mechanism, combining features of two classes: Multimodal2VisionAttention and CLIPSdpaAttention. However, the actual implementation seems to be an empty pass, indicating a potential mistake in the class inheritance.

**Key Implementation Details**

* Implements `Multimodal2VisionAttention`
* Inherites from `CLIPSdpaAttention`

**Notable Dependencies or Requirements**

None specified.

**Primary Purpose**
The primary purpose of this class is not explicitly stated due to its empty implementation, but it likely aims to combine the attention mechanisms of Multimodal2VisionAttention and CLIPSdpaAttention.


key: chroma:document
string_value:
CODE:
# Check that adding the second base class correctly set the parent, even though in clip it does not have the "Vision" part
class Multimodal2VisionFlashAttention2(CLIPFlashAttention2, Multimodal2VisionAttention):


SUMMARY:
**Summary:**

The `Multimodal2VisionFlashAttention2` class is a subclass of both `CLIPFlashAttention2` and `Multimodal2VisionAttention`. This allows inheriting the functionality of `CLIPFlashAttention2` while also utilizing the multimodal attention mechanism from its base class.

**Key Implementation Details:**

- Inherited `CLIBrushLayerNormalization`
- Implemented `Multimodal2VisionFlashAttention.forward()` to utilize the vision module's flash attention
- Utilizes `CLIPFlashAttention2`'s `config` and `num_heads_per_layer` features

**Notable Dependencies or Requirements:**

- `CLIBrushLayerNormalization`
- Extends `CLIPFlashAttention2` via inheritance
- Depends on `Multimodal2VisionAttention` for multimodal attention functionality


key: chroma:document
string_value:
CODE:

class Multimodal2VisionMLP(CLIPMLP):


SUMMARY:
**Summary:**

The `Multimodal2VisionMLP` class inherits from `CLIPMLP`, inheriting its functionality. The actual implementation of the MLMPLP is empty (`pass` statement), indicating that this class likely requires additional configuration or override of parent class methods.

**Key Details:**

* **Method Signature:** None explicitly defined; inheritance from `CLIPMLP` implies method implementation.
* **Design Choice:** Empty class body suggests need for further customization or subclassing.
* **Notable Dependencies/Requirements:** Requires a base class (`CLIPMLP`) and potential configuration options.


key: chroma:document
string_value:
CODE:

class Multimodal2VisionEncoderLayer(CLIPEncoderLayer):
    def __init__(self, config):
        super().__init__()
        self.self_attn = MULTIMODAL2_VISION_ATTENTION_CLASSES[config._attn_implementation](config)


SUMMARY:
**Multimodal 2 Vision Encoder Layer Summary**

This class implements the 2nd layer of the Multimodal 2 Vision encoder, inheriting from CLIP's EncoderLayer. Its primary purpose is to facilitate multi-modal attention and feature transformation.

**Key Components:**

* `self_attn`: Instantiates a self-attention mechanism using `MULTIMODAL2_VISION_ATTENTION_CLASSES` based on `_attn_implementation`.
* `mlp`: Creates an MLP (Multi-Layer Perceptron) instance for feature transformations, utilizing the provided `config`.

**Interfaces and Dependencies:**

* Inherit from `CLIPEncoderLayer`
* Utilizes external class `Multimodal2VisionMLP` 
* depends on `_attn_implementation` configuration value


key: chroma:document
string_value:
CODE:

class Multimodal2VisionEncoder(CLIPEncoder):
    def __init__(self, config):
        super().__init__(config)


SUMMARY:
**Summary:**

The `Multimodal2VisionEncoder` class is a custom encoder that extends `CLIPEncoder`. It inherits the configuration and initialization behavior from its parent class. In the constructor, it:

* Initializes a list of `nn.ModuleList` containing multiple instances of `Multimodal2VisionEncoderLayer`, configured by the provided `config`.
* The number of layers in the model is determined by `config.num_hidden_layers`.

**Key Details:**

* Extends `CLIPEncoder` using inheritance.
* Employs a sequence of `Multimodal2VisionEncoderLayers` with user-configured architecture.

Note: Without more context, it's difficult to pinpoint key algorithms or design choices. However, this summary provides a concise overview of the class's purpose and composition.


key: docstring
string_value:
# Finally here the `Vision` part was correct in CLIP, but we still need to tell it that the encoder arg should use it as well


key: chroma:document
string_value:
CODE:
# Finally here the `Vision` part was correct in CLIP, but we still need to tell it that the encoder arg should use it as well
class Multimodal2VisionTransformer(CLIPVisionTransformer):
    def __init__(self, config):
        super().__init__(config)


SUMMARY:
**Summary**

The `Multimodal2VisionTransformer` class is a subclass of `CLIPVisionTransformer`. It initializes with a `config` object and sets up an instance of `Multimodal2VisionEncoder`, leveraging the `Vision` part from the CLIP architecture to provide multimodal input processing. 

**Key Details**

- Class: `Multimodal2VisionTransformer`
- Subclass of: `CLIPVisionTransformer`
- Depends on: `Multimodal2VisionEncoder`

**Notable Dependencies/Requirements**

- External dependency:
  - No explicit external dependencies mentioned.
  
 However, this appears to depend on the CLIP library or similar. The actual implementation may include additional external dependencies not explicitly listed here.

Note that some parts of this code snippet seem incomplete (e.g., the `Docstring` entry), but these are likely intended for more substantial documentation and not part of the core class functionality.


key: chroma:document
string_value:
CODE:

class Multimodal2VisionPreTrainedModel(CLIPPreTrainedModel):
    def _init_weights(self, module):
        if isinstance(module, Multimodal2VisionMLP):


SUMMARY:
**Summary:**

The `Multimodal2VisionPreTrantedModel` class inherits from `CLIPPreTrainedModel`. The `_init_weights` method checks if the input module is an instance of `Multimodal2VisionMLP`, which is not implemented anywhere in the provided code chunk.

**Key Insights and Requirements:**

- Implemented Interfaces: None (not inheriting or implementing any interfaces)
- External Dependencies:
  - `CLIPPreTrainedModel`
- Notable Design Choices:
  - Overriding `_init_weights` method to explicitly handle `Multimodal2VisionMLP` instances


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_from_uppercase_model.py


key: chroma:document
string_value:
CODE:

class FromUppercaseModelAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(self, config):
        super().__init__()
        self.config = config
        self.embed_dim = config.hidden_size
        self.num_heads = config.num_attention_heads
        self.head_dim = self.embed_dim // self.num_heads
        if self.head_dim * self.num_heads != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`:"
                f" {self.num_heads})."
            )
        self.scale = self.head_dim**-0.5
        self.dropout = config.attention_dropout

        self.k_proj = nn.Linear(self.embed_dim, self.embed_dim)
        self.v_proj = nn.Linear(self.embed_dim, self.embed_dim)
        self.q_proj = nn.Linear(self.embed_dim, self.embed_dim)
        self.out_proj = nn.Linear(self.embed_dim, self.embed_dim)

    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        causal_attention_mask: Optional[torch.Tensor] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        """Input shape: Batch x Time x Channel"""

        bsz, tgt_len, embed_dim = hidden_states.size()

        # get query proj
        query_states = self.q_proj(hidden_states) * self.scale
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

        proj_shape = (bsz * self.num_heads, -1, self.head_dim)
        query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
        key_states = key_states.view(*proj_shape)
        value_states = value_states.view(*proj_shape)

        src_len = key_states.size(1)
        attn_weights = torch.bmm(query_states, key_states.transpose(1, 2))

        if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
            raise ValueError(
                f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
                f" {attn_weights.size()}"
            )

        # apply the causal_attention_mask first
        if causal_attention_mask is not None:
            if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):
                raise ValueError(
                    f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is"
                    f" {causal_attention_mask.size()}"
                )
            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + causal_attention_mask
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        if attention_mask is not None:
            if attention_mask.size() != (bsz, 1, tgt_len, src_len):
                raise ValueError(
                    f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}"
                )
            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        attn_weights = nn.functional.softmax(attn_weights, dim=-1)

        if output_attentions:
            # this operation is a bit akward, but it's required to
            # make sure that attn_weights keeps its gradient.
            # In order to do so, attn_weights have to reshaped
            # twice and have to be reused in the following
            attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
        else:
            attn_weights_reshaped = None

        attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training)

        attn_output = torch.bmm(attn_probs, value_states)

        if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
            raise ValueError(
                f"`attn_output` should be of size {(bsz, self.num_heads, tgt_len, self.head_dim)}, but is"
                f" {attn_output.size()}"
            )

        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
        attn_output = attn_output.transpose(1, 2)
        attn_output = attn_output.reshape(bsz, tgt_len, embed_dim)

        attn_output = self.out_proj(attn_output)



SUMMARY:
**Multi-Head Attention Model**

This is an implementation of the multi-head attention mechanism from the 'Attention Is All You Need' paper.

### Purpose and Functionality

The primary purpose of this model is to compute the weighted sum of the input sequence, based on the similarity between the query and key sequences. The output is a new sequence that represents the weighted sum of all possible inputs at each position.

### Key Implementation Details

*   The model uses a multi-head attention mechanism with `num_heads` heads.
*   Each head applies a linear transformation (projection) to the input sequence using separate linear layers for queries (`q_proj`), keys (`k_proj`), and values (`v_proj`).
*   The output of each head is a tensor with shape `(batch_size \* num_heads, tgt_len, head_dim)`.

### Notable Dependencies or Requirements

*   Requires PyTorch 1.9+
*   Uses the `nn.functional` module for softmax and dropout functions.

### Method Signatures and Interfaces

*   `_shape(tensor: torch.Tensor, seq_len: int, bsz: int) -> torch.Tensor`: A helper function to reshape tensors.
*   `forward(hidden_states: torch.Tensor, ...) -> Tuple[torch.Tensor, Optional[torch.Tensor]]`: The main forward pass method.

### Class Variables and Attributes

*   `config`: An input configuration object that contains hyperparameters such as `hidden_size`, `num_attention_heads`, and `attention_dropout`.
*   `scale`: A scaling factor used to stabilize the attention computation (`self.head_dim**-0.5`).
*   `dropout`: The dropout probability used in the attention mechanism (`config.attention_dropout`).


key: chroma:document
string_value:
CODE:

class FromUppercaseModelFlashAttention2(FromUppercaseModelAttention):
    """
    FromUppercaseModelAttention flash attention module. This module inherits from `FromUppercaseModelAttention` as the weights of the module stays
    untouched. The only required change would be on the forward pass where it needs to correctly call the public API of
    flash attention and deal with padding tokens in case the input contains any of them.
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        # TODO: Should be removed once Flash Attention for RoCm is bumped to 2.1.
        # flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. Reference: https://github.com/Dao-AILab/flash-attention/releases/tag/v2.1.0.
        # Beware that with flash_attn<2.1, using q_seqlen != k_seqlen (except for the case q_seqlen == 1) produces a wrong mask (top-left).
        self._flash_attn_uses_top_left_mask = not is_flash_attn_greater_or_equal_2_10()

    # Adapted from transformers.models.llama.modeling_llama.LlamaFlashAttention2.forward
    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        causal_attention_mask: Optional[torch.Tensor] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        output_attentions = False

        batch_size, q_len, _ = hidden_states.size()

        query_states = self.q_proj(hidden_states)
        key_states = self.k_proj(hidden_states)
        value_states = self.v_proj(hidden_states)

        # Flash attention requires the input to have the shape
        # batch_size x seq_length x head_dim x hidden_dim
        # therefore we just need to keep the original shape
        query_states = query_states.view(batch_size, q_len, self.num_heads, self.head_dim)
        key_states = key_states.view(batch_size, q_len, self.num_heads, self.head_dim)
        value_states = value_states.view(batch_size, q_len, self.num_heads, self.head_dim)

        dropout_rate = self.dropout if self.training else 0.0

        # In PEFT, usually we cast the layer norms in float32 for training stability reasons
        # therefore the input hidden states gets silently casted in float32. Hence, we need
        # cast them back in the correct dtype just to be sure everything works as expected.
        # This might slowdown training & inference so it is recommended to not cast the LayerNorms
        # in fp32.

        input_dtype = query_states.dtype
        if input_dtype == torch.float32:
            if torch.is_autocast_enabled():
                target_dtype = torch.get_autocast_gpu_dtype()
            # Handle the case where the model is quantized
            elif hasattr(self.config, "_pre_quantization_dtype"):
                target_dtype = self.config._pre_quantization_dtype
            else:
                target_dtype = self.q_proj.weight.dtype

            logger.warning_once(
                f"The input hidden states seems to be silently casted in float32, this might be related to"
                f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in"
                f" {target_dtype}."
            )

            query_states = query_states.to(target_dtype)
            key_states = key_states.to(target_dtype)
            value_states = value_states.to(target_dtype)

        attn_output = _flash_attention_forward(
            query_states,
            key_states,
            value_states,
            attention_mask,
            q_len,
            dropout=dropout_rate,
            is_causal=causal_attention_mask is not None,
            use_top_left_mask=self._flash_attn_uses_top_left_mask,
        )

        attn_output = attn_output.reshape(batch_size, q_len, self.embed_dim).contiguous()
        attn_output = self.out_proj(attn_output)

        if not output_attentions:
            attn_weights = None



SUMMARY:
**Class Summary: FromUppercaseModelFlashAttention2**

This class inherits from `FromUppercaseModelAttention` and implements a modified version of the Flash Attention module for compatibility with previous versions.

**Main Purpose and Functionality**
=====================================

The primary purpose of this class is to provide a forward pass through the modified Flash Attention module, which takes into account padding tokens in the input sequence.

**Key Implementation Details**
-----------------------------

* The `forward` method uses the `flash_attention_forward` function from `transformers.models.llama.modeling_llama.LlamaFlashAttention2`.
* It casts the layer norms to a specific data type (float32 or _pre_quantization_dtype) for training stability reasons.
* The `use_top_left_mask` attribute is set based on whether the Flash Attention version is 2.1 or later.

**Notable Dependencies and Requirements**
-----------------------------------------

* `flash_attention` module from `transformers.models.llama.modeling_llama`
* `_flash_attention_forward` function from `transformers.models.llama.modeling_llama.LlamaFlashAttention2`
* `_pre_quantization_dtype` attribute in the config
* Optional tensors for attention mask, causal attention mask, and output attentions

**Method Signatures and Interfaces**
------------------------------------

The class has one method:

* `forward`: takes `hidden_states`, `attention_mask`, `causal_attention_mask`, and `output_attentions` as input, returns a tuple of `(attn_output, attn_weights)`.

**External Dependencies**
------------------------

* The `_flash_attn_uses_top_left_mask` attribute depends on the `is_flash_attn_greater_or_equal_2_10()` function.
* The `_pre_quantization_dtype` attribute depends on the config.


key: chroma:document
string_value:
CODE:

class FromUppercaseModelSdpaAttention(FromUppercaseModelAttention):
    """
    SDPA attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from
    `FromUppercaseModelAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to
    SDPA API.
    """

    # Adapted from FromUppercaseModelAttention.forward
    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        causal_attention_mask: Optional[torch.Tensor] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        if output_attentions:
            # TODO: Improve this warning with e.g. `model.config.attn_implementation = "manual"` once this is implemented.
            logger.warning_once(
                "FromUppercaseModelModel is using FromUppercaseModelSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not "
                "support `output_attentions=True`. Falling back to the manual attention implementation, but specifying "
                "the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can "
                'be removed using the argument `attn_implementation="eager"` when loading the model.'
            )
            return super().forward(
                hidden_states=hidden_states,
                attention_mask=attention_mask,
                causal_attention_mask=causal_attention_mask,
                output_attentions=output_attentions,
            )

        # FROM_UPPERCASE_MODEL text model uses both `causal_attention_mask` and `attention_mask`
        if attention_mask is not None and causal_attention_mask is not None:
            attn_mask = attention_mask + causal_attention_mask
        elif causal_attention_mask is not None:
            attn_mask = causal_attention_mask
        else:
            attn_mask = attention_mask

        bsz, tgt_len, embed_dim = hidden_states.size()

        query_states = self.q_proj(hidden_states)
        key_states = self.k_proj(hidden_states)
        value_states = self.v_proj(hidden_states)

        query_states = query_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)
        key_states = key_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)
        value_states = value_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)

        # SDPA with memory-efficient backend is currently (torch==2.1.2) bugged with non-contiguous inputs with custom attn_mask,
        # Reference: https://github.com/pytorch/pytorch/issues/112577.
        if not is_torch_greater_or_equal_than_2_2 and query_states.device.type == "cuda" and attn_mask is not None:
            query_states = query_states.contiguous()
            key_states = key_states.contiguous()
            value_states = value_states.contiguous()

        # FROM_UPPERCASE_MODEL text model uses both `causal_attention_mask` and `attention_mask` sequentially.
        attn_output = torch.nn.functional.scaled_dot_product_attention(
            query_states,
            key_states,
            value_states,
            attn_mask=attn_mask,
            dropout_p=self.dropout if self.training else 0.0,
            scale=self.scale,
        )

        attn_output = attn_output.transpose(1, 2)
        attn_output = attn_output.reshape(bsz, tgt_len, embed_dim)

        attn_output = self.out_proj(attn_output)



SUMMARY:
**Summary:**

This code defines the `FromUppercaseModelSdpaAttention` class, which inherits from `FromUppercaseModelAttention`. The main purpose is to adapt the attention mechanism of the base model (`torch.nn.functional.scaled_dot_product_attention`) to support SDPA (Scaled-Dot-Product Attention).

**Key Implementation Details:**

* Uses `scaled_dot_product_attention` with modified forward pass to accommodate custom `attn_mask` and `causal_attention_mask`.
* Projects input states using `q_proj`, `k_proj`, and `v_proj`.
* Reshapes projections for scaling dot-product attention.
* Handles compatibility issues between PyTorch 2.x and SDPA backend.
* Applies output projection with `out_proj`.

**Notable Dependencies:**

* `torch.nn.functional.scaled_dot_product_attention`
* Custom `attn_mask` (sum of `attention_mask` and `causal_attention_mask`)
* `FromUppercaseModelAttention`

**Interface:**
 
The class respects the following interfaces:
- `forward(hidden_states, attention_mask, causal_attention_mask, output_attentions)`
- `q_proj`, `k_proj`, and `v_proj` projections
- `out_proj` output projection


key: chroma:document
string_value:
CODE:

class FromUppercaseModelMLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.activation_fn = ACT2FN[config.hidden_act]
        self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
        self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states = self.fc1(hidden_states)
        hidden_states = self.activation_fn(hidden_states)
        hidden_states = self.fc2(hidden_states)


SUMMARY:
**Class Summary: `FromUppercaseModelMLP`**

*   **Purpose:** A PyTorch neural network model that takes input `hidden_states` and applies multilayer perceptron (MLP) transformations, followed by an activation function and an output transformation.
*   **Primary Purpose:** The primary purpose of this class is to implement a simple multilayer perceptron (MLP) for the attention mechanism using a feed-forward neural network (FNN).

**Key Implementation Details:**

*   **Activation Function:** Uses the `ACT2FN` dictionary to select an activation function based on the `config.hidden_act`.
    *   `self.activation_fn = ACT2FN[config.hidden_act]`
*   **Neural Network Layers:**
    1.  Input `hidden_states` is passed through a fully connected (`fc`) layer with weights and biases determined by `config.hidden_size`.
        *   `self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)`
    2.  The output of the first neural network layer undergoes an activation function transformation.
        *   `hidden_states = self.activation_fn(hidden_states)`
    3.  The previous output is then passed through another fully connected (`fc`) layer with weights and biases determined by `config.intermediate_size`.
        *   `self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)`

**Notable Dependencies or Requirements:**

*   **PyTorch Tensors:** Uses PyTorch tensors to represent the input `hidden_states`.
*   **Neural Network Modules:** Depends on PyTorch neural network modules (e.g., `nn.Linear`).
*   **ACT2FN Dictionary:** Utilizes a dictionary (`ACT2FN`) that maps activation function names to their corresponding functions.


key: chroma:document
string_value:
CODE:

class FromUppercaseModelEncoderLayer(nn.Module):
    def __init__(self, config: FromUppercaseModelConfig):
        super().__init__()
        self.embed_dim = config.hidden_size
        self.self_attn = FROM_UPPERCASE_MODEL_ATTENTION_CLASSES[config._attn_implementation](config)
        self.layer_norm1 = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_eps)
        self.mlp = FromUppercaseModelMLP(config)
        self.layer_norm2 = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_eps)

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: torch.Tensor,
        causal_attention_mask: torch.Tensor,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[torch.FloatTensor]:
        """
        Args:
            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
            attention_mask (`torch.FloatTensor`): attention mask of size
                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
                `(config.encoder_attention_heads,)`.
            output_attentions (`bool`, *optional*):
                Whether or not to return the attentions tensors of all attention layers. See `attentions` under
                returned tensors for more detail.
        """
        residual = hidden_states

        hidden_states = self.layer_norm1(hidden_states)
        hidden_states, attn_weights = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            causal_attention_mask=causal_attention_mask,
            output_attentions=output_attentions,
        )
        hidden_states = residual + hidden_states

        residual = hidden_states
        hidden_states = self.layer_norm2(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

        outputs = (hidden_states,)

        if output_attentions:
            outputs += (attn_weights,)



SUMMARY:
**Summary**

The `FromUppercaseModelEncoderLayer` class is an encoder layer in the Transformer architecture. It implements self-attention mechanism and follows residual connected feed-forward network (RCFFNN) architecture.

**Key Implementation Details:**

*   **Self-Attention Mechanism**: Utilizes `self_attn` which is an instance of `FROM_UPPERCASE_MODEL_ATTENTION_CLASSES[config]._attn_implementation](config)` from `FromUppercaseModelATTENTION_CLASSES`. It takes in `hidden_states`, `attention_mask`, and `causal_attention_mask`.
*   **Residual Connections**: Applies residual connections after two separate self-attention mechanisms, which enhance stability and make the network easier to train.
*   **MLP (Multi-Layer Perceptron) Block**: Implemented using an instance of `FromUppercaseModelMLPB` with configuration settings from `config`.

**Notable Dependencies:**

*   Importantly uses `nn.Module`, `torch.FloatTensor`, `optional` type hinting and `Trapeze` functions
*   Relies on `FROM_UPPERCASE_MODEL_ATTENTION_CLASSES[config._attn_implementation]`
*   Incorporates `FromUppercaseModelMLP` with configuration settings from `config`.

**Other Requirements:**

- Uses Torch (PyTorch) library (`torch.Tensor`)
 - Implements LayerNorm (`self.layer_norm1`, `self.layer_norm2`)


key: chroma:document
string_value:
CODE:

def get_setup_file():
    parser = argparse.ArgumentParser()
    parser.add_argument("-f")
    args = parser.parse_args()


SUMMARY:
**Summary:**

The `get_setup_file` function is used to retrieve the setup file name from command-line arguments. It utilizes the `argparse` library to parse the command-line input and extract the value of the `-f` (or `--file`) option.

**Key Details:**

* The function uses `argparse.ArgumentParser()` to create a parser.
* Adds a single argument `-f` with no default value.
* Uses `parser.parse_args()` to parse the command-line arguments and extract the value of `-f`.
* Returns the extracted file name as a string (`return args.f`).

**Notable Dependencies:**

* `argparse`: Used for command-line argument parsing.

Note: No external dependencies or interfaces are explicitly stated in this function. However, it assumes that the `argparse` library is available and properly configured in the environment.


key: chroma:document
string_value:
CODE:

def get_results(output_dir):
    results = {}
    path = os.path.join(output_dir, "all_results.json")
    if os.path.exists(path):
        with open(path, "r") as f:
            results = json.load(f)
    else:
        raise ValueError(f"can't find {path}")


SUMMARY:
**Function Summary:**

`get_results(output_dir)` retrieves stored JSON results from the specified output directory.

**Key Implementation Details:**

* Reads and returns existing "all_results.json" file from the provided `output_dir`.
* If file does not exist, raises a ValueError.

**Notable Dependencies:**

* Relies on `json` module for parsing the JSON file.
* Utilizes `os.path.join()` to construct file path.
* Inherent usage of built-in file I/O functions (e.g., `open()`, `with` statement).

No external dependencies are explicitly stated in this snippet. However, the `json` module is a fundamental Python standard library dependency.


key: chroma:document
string_value:
CODE:

def is_cuda_available():


SUMMARY:
**Summary:**

The `is_cuda_available` function checks if CUDA (NVIDIA GPU acceleration) is available on the system by using TensorFlow's configuration API (`tf.config.list_physical_devices("GPU")`) and returns a boolean value indicating availability.

**Implementation Details:**

* The primary purpose of this function is to determine if the system has a CUDA-compatible GPU.
* It uses the `list_physical_devices` method from TensorFlow's configuration API to query for physical devices with the "GPU" label.

**Key Information:**

* Returns: `bool` indicating whether CUDA (GPU) is available
* Uses: TensorFlow's `tf.config.list_physical_devices("GPU")`
* Inherited/Implemented Interfaces: None


key: chroma:document
string_value:
CODE:

class ExamplesTests(TestCasePlus):
    @skip("Skipping until shape inference for to_tf_dataset PR is merged.")
    def test_run_text_classification(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_text_classification.py
            --model_name_or_path distilbert/distilbert-base-uncased
            --output_dir {tmp_dir}
            --overwrite_output_dir
            --train_file ./tests/fixtures/tests_samples/MRPC/train.csv
            --validation_file ./tests/fixtures/tests_samples/MRPC/dev.csv
            --do_train
            --do_eval
            --per_device_train_batch_size=2
            --per_device_eval_batch_size=1
            --learning_rate=1e-4
            --max_steps=10
            --warmup_steps=2
            --seed=42
            --max_seq_length=128
            """.split()

        if is_cuda_available():
            testargs.append("--fp16")

        with patch.object(sys, "argv", testargs):
            run_text_classification.main()
            # Reset the mixed precision policy so we don't break other tests
            keras.mixed_precision.set_global_policy("float32")
            result = get_results(tmp_dir)
            self.assertGreaterEqual(result["eval_accuracy"], 0.75)

    def test_run_clm(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_clm.py
            --model_name_or_path distilbert/distilgpt2
            --train_file ./tests/fixtures/sample_text.txt
            --validation_file ./tests/fixtures/sample_text.txt
            --do_train
            --do_eval
            --block_size 128
            --per_device_train_batch_size 2
            --per_device_eval_batch_size 1
            --num_train_epochs 2
            --output_dir {tmp_dir}
            --overwrite_output_dir
            """.split()

        if len(tf.config.list_physical_devices("GPU")) > 1:
            # Skipping because there are not enough batches to train the model + would need a drop_last to work.
            return

        with patch.object(sys, "argv", testargs):
            run_clm.main()
            result = get_results(tmp_dir)
            self.assertLess(result["eval_perplexity"], 100)

    def test_run_mlm(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_mlm.py
            --model_name_or_path distilbert/distilroberta-base
            --train_file ./tests/fixtures/sample_text.txt
            --validation_file ./tests/fixtures/sample_text.txt
            --max_seq_length 64
            --output_dir {tmp_dir}
            --overwrite_output_dir
            --do_train
            --do_eval
            --prediction_loss_only
            --num_train_epochs=1
            --learning_rate=1e-4
        """.split()

        with patch.object(sys, "argv", testargs):
            run_mlm.main()
            result = get_results(tmp_dir)
            self.assertLess(result["eval_perplexity"], 42)

    def test_run_ner(self):
        # with so little data distributed training needs more epochs to get the score on par with 0/1 gpu
        epochs = 7 if get_gpu_count() > 1 else 2

        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_ner.py
            --model_name_or_path google-bert/bert-base-uncased
            --train_file tests/fixtures/tests_samples/conll/sample.json
            --validation_file tests/fixtures/tests_samples/conll/sample.json
            --output_dir {tmp_dir}
            --overwrite_output_dir
            --do_train
            --do_eval
            --warmup_steps=2
            --learning_rate=2e-4
            --per_device_train_batch_size=2
            --per_device_eval_batch_size=2
            --num_train_epochs={epochs}
            --seed 7
        """.split()

        with patch.object(sys, "argv", testargs):
            run_ner.main()
            result = get_results(tmp_dir)
            self.assertGreaterEqual(result["accuracy"], 0.75)

    def test_run_squad(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_qa.py
            --model_name_or_path google-bert/bert-base-uncased
            --version_2_with_negative
            --train_file tests/fixtures/tests_samples/SQUAD/sample.json
            --validation_file tests/fixtures/tests_samples/SQUAD/sample.json
            --output_dir {tmp_dir}
            --overwrite_output_dir
            --max_steps=10
            --warmup_steps=2
            --do_train
            --do_eval
            --learning_rate=2e-4
            --per_device_train_batch_size=2
            --per_device_eval_batch_size=1
        """.split()

        with patch.object(sys, "argv", testargs):
            run_squad.main()
            result = get_results(tmp_dir)
            self.assertGreaterEqual(result["f1"], 30)
            self.assertGreaterEqual(result["exact"], 30)

    def test_run_swag(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_swag.py
            --model_name_or_path google-bert/bert-base-uncased
            --train_file tests/fixtures/tests_samples/swag/sample.json
            --validation_file tests/fixtures/tests_samples/swag/sample.json
            --output_dir {tmp_dir}
            --overwrite_output_dir
            --max_steps=20
            --warmup_steps=2
            --do_train
            --do_eval
            --learning_rate=2e-4
            --per_device_train_batch_size=2
            --per_device_eval_batch_size=1
        """.split()

        with patch.object(sys, "argv", testargs):
            run_swag.main()
            result = get_results(tmp_dir)
            self.assertGreaterEqual(result["val_accuracy"], 0.8)

    @slow
    def test_run_summarization(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_summarization.py
            --model_name_or_path google-t5/t5-small
            --train_file tests/fixtures/tests_samples/xsum/sample.json
            --validation_file tests/fixtures/tests_samples/xsum/sample.json
            --output_dir {tmp_dir}
            --overwrite_output_dir
            --max_steps=50
            --warmup_steps=8
            --do_train
            --do_eval
            --learning_rate=2e-4
            --per_device_train_batch_size=2
            --per_device_eval_batch_size=1
        """.split()

        with patch.object(sys, "argv", testargs):
            run_summarization.main()
            result = get_results(tmp_dir)
            self.assertGreaterEqual(result["rouge1"], 10)
            self.assertGreaterEqual(result["rouge2"], 2)
            self.assertGreaterEqual(result["rougeL"], 7)
            self.assertGreaterEqual(result["rougeLsum"], 7)

    @slow
    def test_run_translation(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_translation.py
            --model_name_or_path Rocketknight1/student_marian_en_ro_6_1
            --source_lang en
            --target_lang ro
            --train_file tests/fixtures/tests_samples/wmt16/sample.json
            --validation_file tests/fixtures/tests_samples/wmt16/sample.json
            --output_dir {tmp_dir}
            --overwrite_output_dir
            --warmup_steps=8
            --do_train
            --do_eval
            --learning_rate=3e-3
            --num_train_epochs 12
            --per_device_train_batch_size=2
            --per_device_eval_batch_size=1
            --source_lang en_XX
            --target_lang ro_RO
        """.split()

        with patch.object(sys, "argv", testargs):
            run_translation.main()
            result = get_results(tmp_dir)
            self.assertGreaterEqual(result["bleu"], 30)

    def test_run_image_classification(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
        testargs = f"""
            run_image_classification.py
            --dataset_name hf-internal-testing/cats_vs_dogs_sample
            --trust_remote_code
            --model_name_or_path microsoft/resnet-18
            --do_train
            --do_eval
            --learning_rate 1e-4
            --per_device_train_batch_size 2
            --per_device_eval_batch_size 1
            --output_dir {tmp_dir}
            --overwrite_output_dir
            --dataloader_num_workers 16
            --num_train_epochs 2
            --train_val_split 0.1
            --seed 42
            --ignore_mismatched_sizes True
            """.split()

        with patch.object(sys, "argv", testargs):
            run_image_classification.main()
            result = get_results(tmp_dir)


SUMMARY:
This is a Python class that implements a set of unit tests for various natural language processing (NLP) tasks using the Hugging Face Transformers library. The tests are designed to verify the correct behavior of different NLP models, such as text classification, language modeling, named entity recognition, question answering, sentiment analysis, and image classification.

The class `ExamplesTests` inherits from a base class not shown in the code snippet, which likely provides a set of common test fixtures or utilities. Each method in this class corresponds to a specific NLP task, and it performs the following steps:

1. It creates a temporary directory (`tmp_dir`) where all the models will be stored.
2. It defines a list of `testargs` that specify the model name, training parameters, and other settings for each test case.
3. It patches the `sys.argv` attribute with the `testargs` so that when running the corresponding script (e.g., `run_text_classification.py`, etc.), it runs with these specific arguments.
4. It calls the `main()` function of the NLP model's script to run the training loop, and then it retrieves the evaluation metrics from the results directory (`tmp_dir`).

The tests are annotated with additional metadata, such as `@slow` or `@test_run-swag`, which may indicate that these tests take longer to execute or require specific execution environments. The tests also include assertions that verify specific values of the evaluation metrics.

Overall, this class provides a comprehensive set of tests for various NLP tasks, ensuring that different models behave correctly under different conditions and training parameters.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/text-classification/run_text_classification.py


key: chroma:document
string_value:
CODE:
# region Helper classes
class SavePretrainedCallback(keras.callbacks.Callback):
    # Hugging Face models have a save_pretrained() method that saves both the weights and the necessary
    # metadata to allow them to be loaded as a pretrained model in future. This is a simple Keras callback
    # that saves the model with this method after each epoch.
    def __init__(self, output_dir, **kwargs):
        super().__init__()
        self.output_dir = output_dir

    def on_epoch_end(self, epoch, logs=None):


SUMMARY:
**SavePretrainedCallback Class Summary**

### Purpose and Functionality

The `SavePretrainedCallback` class is a Keras callback that saves Hugging Face models using the `save_pretrained()` method, which includes model weights and metadata, after each epoch.

### Key Implementation Details

* Saves the model using `model.save_pretrained(output_dir)` after each epoch.
* Uses the `__init__()` method to initialize the output directory.

### Notable Dependencies or Requirements

* Keras
* Hugging Face models (inherited from `Docstring`)

Note: The `Docstring` class is not shown in this code snippet, but it is likely a base class for other callback classes used by Hugging Face models.


key: chroma:document
string_value:
CODE:

def main():
    # region Argument parsing
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_text_classification", model_args, data_args, framework="tensorflow")

    output_dir = Path(training_args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)
    # endregion

    # region Checkpoints
    # Detecting last checkpoint.
    checkpoint = None
    if len(os.listdir(training_args.output_dir)) > 0 and not training_args.overwrite_output_dir:
        if (output_dir / CONFIG_NAME).is_file() and (output_dir / TF2_WEIGHTS_NAME).is_file():
            checkpoint = output_dir
            logger.info(
                f"Checkpoint detected, resuming training from checkpoint in {training_args.output_dir}. To avoid this"
                " behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )
        else:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to continue regardless."
            )

    # endregion

    # region Logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO)

    logger.info(f"Training/evaluation parameters {training_args}")
    # endregion

    # region Loading data
    # For CSV/JSON files, this script will use the 'label' field as the label and the 'sentence1' and optionally
    # 'sentence2' fields as inputs if they exist. If not, the first two fields not named label are used if at least two
    # columns are provided. Note that the term 'sentence' can be slightly misleading, as they often contain more than
    # a single grammatical sentence, when the task requires it.
    #
    # If the CSVs/JSONs contain only one non-label column, the script does single sentence classification on this
    # single column. You can easily tweak this behavior (see below)
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    data_files = {"train": data_args.train_file, "validation": data_args.validation_file, "test": data_args.test_file}
    data_files = {key: file for key, file in data_files.items() if file is not None}

    for key in data_files.keys():
        logger.info(f"Loading a local file for {key}: {data_files[key]}")

    if data_args.input_file_extension == "csv":
        # Loading a dataset from local csv files
        datasets = load_dataset(
            "csv",
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
        )
    else:
        # Loading a dataset from local json files
        datasets = load_dataset("json", data_files=data_files, cache_dir=model_args.cache_dir)
    # See more about loading any type of standard or custom dataset at
    # https://huggingface.co/docs/datasets/loading_datasets.
    # endregion

    # region Label preprocessing
    # If you've passed us a training set, we try to infer your labels from it
    if "train" in datasets:
        # By default we assume that if your label column looks like a float then you're doing regression,
        # and if not then you're doing classification. This is something you may want to change!
        is_regression = datasets["train"].features["label"].dtype in ["float32", "float64"]
        if is_regression:
            num_labels = 1
        else:
            # A useful fast method:
            # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.unique
            label_list = datasets["train"].unique("label")
            label_list.sort()  # Let's sort it for determinism
            num_labels = len(label_list)
    # If you haven't passed a training set, we read label info from the saved model (this happens later)
    else:
        num_labels = None
        label_list = None
        is_regression = None
    # endregion

    # region Load model config and tokenizer
    if checkpoint is not None:
        config_path = training_args.output_dir
    elif model_args.config_name:
        config_path = model_args.config_name
    else:
        config_path = model_args.model_name_or_path
    if num_labels is not None:
        config = AutoConfig.from_pretrained(
            config_path,
            num_labels=num_labels,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        config = AutoConfig.from_pretrained(
            config_path,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    # endregion

    # region Dataset preprocessing
    # Again, we try to have some nice defaults but don't hesitate to tweak to your use case.
    column_names = {col for cols in datasets.column_names.values() for col in cols}
    non_label_column_names = [name for name in column_names if name != "label"]
    if "sentence1" in non_label_column_names and "sentence2" in non_label_column_names:
        sentence1_key, sentence2_key = "sentence1", "sentence2"
    elif "sentence1" in non_label_column_names:
        sentence1_key, sentence2_key = "sentence1", None
    else:
        if len(non_label_column_names) >= 2:
            sentence1_key, sentence2_key = non_label_column_names[:2]
        else:
            sentence1_key, sentence2_key = non_label_column_names[0], None

    if data_args.max_seq_length > tokenizer.model_max_length:
        logger.warning(
            f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the "
            f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
        )
    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    # Ensure that our labels match the model's, if it has some pre-specified
    if "train" in datasets:
        if not is_regression and config.label2id != PretrainedConfig(num_labels=num_labels).label2id:
            label_name_to_id = config.label2id
            if sorted(label_name_to_id.keys()) == sorted(label_list):
                label_to_id = label_name_to_id  # Use the model's labels
            else:
                logger.warning(
                    "Your model seems to have been trained with labels, but they don't match the dataset: "
                    f"model labels: {sorted(label_name_to_id.keys())}, dataset labels:"
                    f" {sorted(label_list)}.\nIgnoring the model labels as a result.",
                )
                label_to_id = {v: i for i, v in enumerate(label_list)}
        elif not is_regression:
            label_to_id = {v: i for i, v in enumerate(label_list)}
        else:
            label_to_id = None
        # Now we've established our label2id, let's overwrite the model config with it.
        config.label2id = label_to_id
        if config.label2id is not None:
            config.id2label = {id: label for label, id in label_to_id.items()}
        else:
            config.id2label = None
    else:
        label_to_id = config.label2id  # Just load the data from the model

    if "validation" in datasets and config.label2id is not None:
        validation_label_list = datasets["validation"].unique("label")
        for val_label in validation_label_list:
            assert val_label in label_to_id, f"Label {val_label} is in the validation set but not the training set!"

    def preprocess_function(examples):
        # Tokenize the texts
        args = (
            (examples[sentence1_key],) if sentence2_key is None else (examples[sentence1_key], examples[sentence2_key])
        )
        result = tokenizer(*args, max_length=max_seq_length, truncation=True)

        # Map labels to IDs
        if config.label2id is not None and "label" in examples:
            result["label"] = [(config.label2id[l] if l != -1 else -1) for l in examples["label"]]
        return result

    datasets = datasets.map(preprocess_function, batched=True, load_from_cache_file=not data_args.overwrite_cache)

    # endregion

    with training_args.strategy.scope():
        # region Load pretrained model
        # Set seed before initializing model
        set_seed(training_args.seed)
        #
        # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently
        # download model & vocab.
        if checkpoint is None:
            model_path = model_args.model_name_or_path
        else:
            model_path = checkpoint
        model = TFAutoModelForSequenceClassification.from_pretrained(
            model_path,
            config=config,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
        # endregion

        # region Convert data to a tf.data.Dataset
        dataset_options = tf.data.Options()
        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF
        num_replicas = training_args.strategy.num_replicas_in_sync

        tf_data = {}
        max_samples = {
            "train": data_args.max_train_samples,
            "validation": data_args.max_val_samples,
            "test": data_args.max_test_samples,
        }
        for key in ("train", "validation", "test"):
            if key not in datasets:
                tf_data[key] = None
                continue
            if (
                (key == "train" and not training_args.do_train)
                or (key == "validation" and not training_args.do_eval)
                or (key == "test" and not training_args.do_predict)
            ):
                tf_data[key] = None
                continue
            if key in ("train", "validation"):
                assert "label" in datasets[key].features, f"Missing labels from {key} data!"
            if key == "train":
                shuffle = True
                batch_size = training_args.per_device_train_batch_size * num_replicas
            else:
                shuffle = False
                batch_size = training_args.per_device_eval_batch_size * num_replicas
            samples_limit = max_samples[key]
            dataset = datasets[key]
            if samples_limit is not None:
                dataset = dataset.select(range(samples_limit))

            # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
            # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
            # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
            # yourself if you use this method, whereas they are automatically inferred from the model input names when
            # using model.prepare_tf_dataset()
            # For more info see the docs:
            # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
            # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset

            data = model.prepare_tf_dataset(
                dataset,
                shuffle=shuffle,
                batch_size=batch_size,
                tokenizer=tokenizer,
            )
            data = data.with_options(dataset_options)
            tf_data[key] = data
        # endregion

        # region Optimizer, loss and compilation

        if training_args.do_train:
            num_train_steps = len(tf_data["train"]) * training_args.num_train_epochs
            if training_args.warmup_steps > 0:
                num_warmup_steps = training_args.warmup_steps
            elif training_args.warmup_ratio > 0:
                num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
            else:
                num_warmup_steps = 0

            optimizer, schedule = create_optimizer(
                init_lr=training_args.learning_rate,
                num_train_steps=num_train_steps,
                num_warmup_steps=num_warmup_steps,
                adam_beta1=training_args.adam_beta1,
                adam_beta2=training_args.adam_beta2,
                adam_epsilon=training_args.adam_epsilon,
                weight_decay_rate=training_args.weight_decay,
                adam_global_clipnorm=training_args.max_grad_norm,
            )
        else:
            optimizer = "sgd"  # Just use any default
        if is_regression:
            metrics = []
        else:
            metrics = ["accuracy"]
        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, metrics=metrics)
        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            push_to_hub_model_id = f"{model_name}-finetuned-text-classification"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "text-classification"}

        if training_args.push_to_hub:
            callbacks = [
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            ]
        else:
            callbacks = []
        # endregion

        # region Training and validation
        if tf_data["train"] is not None:
            model.fit(
                tf_data["train"],
                validation_data=tf_data["validation"],
                epochs=int(training_args.num_train_epochs),
                callbacks=callbacks,
            )
        if tf_data["validation"] is not None:
            logger.info("Computing metrics on validation data...")
            if is_regression:
                loss = model.evaluate(tf_data["validation"])
                logger.info(f"Eval loss: {loss:.5f}")
            else:
                loss, accuracy = model.evaluate(tf_data["validation"])
                logger.info(f"Eval loss: {loss:.5f}, Eval accuracy: {accuracy * 100:.4f}%")
            if training_args.output_dir is not None:
                output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
                eval_dict = {"eval_loss": loss}
                if not is_regression:
                    eval_dict["eval_accuracy"] = accuracy
                with open(output_eval_file, "w") as writer:
                    writer.write(json.dumps(eval_dict))
        # endregion

        # region Prediction
        if tf_data["test"] is not None:
            logger.info("Doing predictions on test dataset...")
            predictions = model.predict(tf_data["test"])["logits"]
            predicted_class = np.squeeze(predictions) if is_regression else np.argmax(predictions, axis=1)
            output_test_file = os.path.join(training_args.output_dir, "test_results.txt")
            with open(output_test_file, "w") as writer:
                writer.write("index\tprediction\n")
                for index, item in enumerate(predicted_class):
                    if is_regression:
                        writer.write(f"{index}\t{item:3.3f}\n")
                    else:
                        item = config.id2label[item]
                        writer.write(f"{index}\t{item}\n")
            logger.info(f"Wrote predictions to {output_test_file}!")
        # endregion

        if training_args.output_dir is not None and not training_args.push_to_hub:
            # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
Here's a concise summary of the provided code:

**Overview**

This is a Python script that trains a transformer-based text classification model using the Hugging Face Transformers library. The script performs various tasks, including data preparation, model compilation, training, validation, and prediction.

**Key Components**

1. **Data Preparation**: Loads dataset from Hugging Face datasets or custom database, preprocesses text data (e.g., tokenization), and divides data into training, validation, and testing sets.
2. **Model Creation**: Initializes the transformer-based model with a specified architecture, learning rate, number of epochs, and other hyperparameters.
3. **Model Compilation**: Compiles the model with an optimizer, loss function, and metrics (e.g., accuracy).
4. **Training**: Trains the model on the training data using the `fit()` method and performs validation on the validation data after each epoch.
5. **Validation and Prediction**: Evaluates the model's performance on the validation data and predicts labels or scores for the testing data.
6. **Prediction Output**: Saves predictions to a text file and writes results to a JSON file.

**Configuration**

The script takes several configuration options as input:

1. **Model Architecture**: Specify the transformer-based architecture (e.g., `bert-base-uncased`, `distilbert-v1`) and learning rate.
2. **Number of Epochs**: Set the number of training epochs.
3. **Batch Size**: Control the batch size for training and prediction.
4. **Output Directory**: Define a directory to save model weights, prediction results, and logs.

**Libraries**

The script relies on several libraries:

1. **Hugging Face Transformers**: For transformer-based architectures, tokenization, and other functionality.
2. **PyTorch**: For building the model and training loop.
3. **NLTK**: For text preprocessing (e.g., tokenization).

Overall, this script provides a flexible framework for training and evaluating text classification models using transformer architectures.


key: chroma:document
string_value:
CODE:

def main():
    # region Argument parsing
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_glue", model_args, data_args, framework="tensorflow")

    if not (training_args.do_train or training_args.do_eval or training_args.do_predict):
        exit("Must specify at least one of --do_train, --do_eval or --do_predict!")
    # endregion

    # region Checkpoints
    checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        checkpoint = get_last_checkpoint(training_args.output_dir)
        if checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif checkpoint is not None and training_args.resume_from_checkpoint is None:
            logger.info(
                f"Checkpoint detected, resuming training at {checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )
    # endregion

    # region Logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    logger.info(f"Training/evaluation parameters {training_args}")
    # endregion

    # region Dataset and labels
    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Downloading and loading a dataset from the hub. In distributed training, the load_dataset function guarantee
    # that only one local process can concurrently download the dataset.
    datasets = load_dataset(
        "nyu-mll/glue",
        data_args.task_name,
        cache_dir=model_args.cache_dir,
        token=model_args.token,
    )
    # See more about loading any type of standard or custom dataset at
    # https://huggingface.co/docs/datasets/loading_datasets.

    is_regression = data_args.task_name == "stsb"
    if not is_regression:
        label_list = datasets["train"].features["label"].names
        num_labels = len(label_list)
    else:
        num_labels = 1

    if data_args.predict_file is not None:
        logger.info("Preparing user-supplied file for predictions...")

        data_files = {"data": data_args.predict_file}

        for key in data_files.keys():
            logger.info(f"Loading a local file for {key}: {data_files[key]}")

        if data_args.predict_file.endswith(".csv"):
            # Loading a dataset from local csv files
            user_dataset = load_dataset("csv", data_files=data_files, cache_dir=model_args.cache_dir)
        else:
            # Loading a dataset from local json files
            user_dataset = load_dataset("json", data_files=data_files, cache_dir=model_args.cache_dir)
        needed_keys = task_to_keys[data_args.task_name]
        for key in needed_keys:
            assert key in user_dataset["data"].features, f"Your supplied predict_file is missing the {key} key!"
        datasets["user_data"] = user_dataset["data"]
    # endregion

    # region Load model config and tokenizer
    #
    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        num_labels=num_labels,
        finetuning_task=data_args.task_name,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast_tokenizer,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    # endregion

    # region Dataset preprocessing
    sentence1_key, sentence2_key = task_to_keys[data_args.task_name]

    # Padding strategy
    if data_args.pad_to_max_length:
        padding = "max_length"
    else:
        # We will pad later, dynamically at batch creation, to the max sequence length in each batch
        padding = False

    # Some models have set the order of the labels to use, so let's make sure we do use it.
    label_to_id = None
    if config.label2id != PretrainedConfig(num_labels=num_labels).label2id and not is_regression:
        # Some have all caps in their config, some don't.
        label_name_to_id = {k.lower(): v for k, v in config.label2id.items()}
        if sorted(label_name_to_id.keys()) == sorted(label_list):
            label_to_id = {i: int(label_name_to_id[label_list[i]]) for i in range(num_labels)}
        else:
            logger.warning(
                "Your model seems to have been trained with labels, but they don't match the dataset: "
                f"model labels: {sorted(label_name_to_id.keys())}, dataset labels: {sorted(label_list)}."
                "\nIgnoring the model labels as a result.",
            )
            label_to_id = {label: i for i, label in enumerate(label_list)}
    if label_to_id is not None:
        config.label2id = label_to_id
        config.id2label = {id: label for label, id in config.label2id.items()}
    elif data_args.task_name is not None and not is_regression:
        config.label2id = {l: i for i, l in enumerate(label_list)}
        config.id2label = {id: label for label, id in config.label2id.items()}

    if data_args.max_seq_length > tokenizer.model_max_length:
        logger.warning(
            f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the "
            f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
        )
    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    def preprocess_function(examples):
        # Tokenize the texts
        args = (
            (examples[sentence1_key],) if sentence2_key is None else (examples[sentence1_key], examples[sentence2_key])
        )
        result = tokenizer(*args, padding=padding, max_length=max_seq_length, truncation=True)

        return result

    datasets = datasets.map(preprocess_function, batched=True, load_from_cache_file=not data_args.overwrite_cache)

    if data_args.pad_to_max_length:
        data_collator = DefaultDataCollator(return_tensors="np")
    else:
        data_collator = DataCollatorWithPadding(tokenizer, return_tensors="np")
    # endregion

    # region Metric function
    metric = evaluate.load("glue", data_args.task_name, cache_dir=model_args.cache_dir)

    def compute_metrics(preds, label_ids):
        preds = preds["logits"]
        preds = np.squeeze(preds) if is_regression else np.argmax(preds, axis=1)
        result = metric.compute(predictions=preds, references=label_ids)
        if len(result) > 1:
            result["combined_score"] = np.mean(list(result.values())).item()
        return result

    # endregion

    with training_args.strategy.scope():
        # region Load pretrained model
        if checkpoint is None:
            model_path = model_args.model_name_or_path
        else:
            model_path = checkpoint
        model = TFAutoModelForSequenceClassification.from_pretrained(
            model_path,
            config=config,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
        # endregion

        # region Convert data to a tf.data.Dataset
        dataset_options = tf.data.Options()
        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF
        num_replicas = training_args.strategy.num_replicas_in_sync

        tf_data = {}
        max_samples = {
            "train": data_args.max_train_samples,
            "validation": data_args.max_eval_samples,
            "validation_matched": data_args.max_eval_samples,
            "validation_mismatched": data_args.max_eval_samples,
            "test": data_args.max_predict_samples,
            "test_matched": data_args.max_predict_samples,
            "test_mismatched": data_args.max_predict_samples,
            "user_data": None,
        }
        for key in datasets.keys():
            if key == "train" or key.startswith("validation"):
                assert "label" in datasets[key].features, f"Missing labels from {key} data!"
            if key == "train":
                shuffle = True
                batch_size = training_args.per_device_train_batch_size * num_replicas
            else:
                shuffle = False
                batch_size = training_args.per_device_eval_batch_size * num_replicas
            samples_limit = max_samples[key]
            dataset = datasets[key]
            if samples_limit is not None:
                dataset = dataset.select(range(samples_limit))

            # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
            # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
            # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
            # yourself if you use this method, whereas they are automatically inferred from the model input names when
            # using model.prepare_tf_dataset()
            # For more info see the docs:
            # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
            # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset
            data = model.prepare_tf_dataset(
                dataset,
                shuffle=shuffle,
                batch_size=batch_size,
                collate_fn=data_collator,
                tokenizer=tokenizer,
            )
            data = data.with_options(dataset_options)
            tf_data[key] = data
        # endregion

        # region Optimizer, loss and compilation
        if training_args.do_train:
            num_train_steps = len(tf_data["train"]) * training_args.num_train_epochs
            if training_args.warmup_steps > 0:
                num_warmup_steps = training_args.warmup_steps
            elif training_args.warmup_ratio > 0:
                num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
            else:
                num_warmup_steps = 0

            optimizer, schedule = create_optimizer(
                init_lr=training_args.learning_rate,
                num_train_steps=num_train_steps,
                num_warmup_steps=num_warmup_steps,
                adam_beta1=training_args.adam_beta1,
                adam_beta2=training_args.adam_beta2,
                adam_epsilon=training_args.adam_epsilon,
                weight_decay_rate=training_args.weight_decay,
                adam_global_clipnorm=training_args.max_grad_norm,
            )
        else:
            optimizer = "sgd"  # Just write anything because we won't be using it
        if is_regression:
            metrics = []
        else:
            metrics = ["accuracy"]
        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, metrics=metrics, jit_compile=training_args.xla)
        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            push_to_hub_model_id = f"{model_name}-finetuned-glue"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "text-classification"}
        model_card_kwargs["task_name"] = data_args.task_name

        if training_args.push_to_hub:
            callbacks = [
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            ]
        else:
            callbacks = []
        # endregion

        # region Training and validation
        if training_args.do_train:
            if training_args.do_eval and not data_args.task_name == "mnli":
                # Do both evaluation and training in the Keras fit loop, unless the task is MNLI
                # because MNLI has two validation sets
                validation_data = tf_data["validation"]
            else:
                validation_data = None
            model.fit(
                tf_data["train"],
                validation_data=validation_data,
                epochs=int(training_args.num_train_epochs),
                callbacks=callbacks,
            )
        # endregion

        # region Evaluation
        if training_args.do_eval:
            # We normally do validation as part of the Keras fit loop, but we run it independently
            # if there was no fit() step (because we didn't train the model) or if the task is MNLI,
            # because MNLI has a separate validation-mismatched validation set

            # In this example, we compute advanced metrics only at the end of training, and only compute
            # loss and accuracy on the validation set each epoch, but
            # if you'd like to compute metrics every epoch that are too complex to be written as
            # standard Keras metrics, you can use our KerasMetricCallback. See
            # https://huggingface.co/docs/transformers/main/en/main_classes/keras_callbacks
            logger.info("*** Evaluate ***")

            # Loop to handle MNLI double evaluation (matched, mis-matched)
            if data_args.task_name == "mnli":
                tasks = ["mnli", "mnli-mm"]
                tf_datasets = [tf_data["validation_matched"], tf_data["validation_mismatched"]]
                raw_datasets = [datasets["validation_matched"], datasets["validation_mismatched"]]
            else:
                tasks = [data_args.task_name]
                tf_datasets = [tf_data["validation"]]
                raw_datasets = [datasets["validation"]]

            for raw_dataset, tf_dataset, task in zip(raw_datasets, tf_datasets, tasks):
                eval_predictions = model.predict(tf_dataset)
                eval_metrics = compute_metrics(eval_predictions, raw_dataset["label"])
                print(f"Evaluation metrics ({task}):")
                print(eval_metrics)
                if training_args.output_dir is not None:
                    output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
                    with open(output_eval_file, "w") as writer:
                        writer.write(json.dumps(eval_metrics))

        # endregion

        # region Prediction
        if training_args.do_predict or data_args.predict_file:
            logger.info("*** Predict ***")

            # Loop to handle MNLI double evaluation (matched, mis-matched)
            tasks = []
            tf_datasets = []
            raw_datasets = []
            if training_args.do_predict:
                if data_args.task_name == "mnli":
                    tasks.extend(["mnli", "mnli-mm"])
                    tf_datasets.extend([tf_data["test_matched"], tf_data["test_mismatched"]])
                    raw_datasets.extend([datasets["test_matched"], datasets["test_mismatched"]])
                else:
                    tasks.append(data_args.task_name)
                    tf_datasets.append(tf_data["test"])
                    raw_datasets.append(datasets["test"])
            if data_args.predict_file:
                tasks.append("user_data")
                tf_datasets.append(tf_data["user_data"])
                raw_datasets.append(datasets["user_data"])

            for raw_dataset, tf_dataset, task in zip(raw_datasets, tf_datasets, tasks):
                test_predictions = model.predict(tf_dataset)
                if "label" in raw_dataset:
                    test_metrics = compute_metrics(test_predictions, raw_dataset["label"])
                    print(f"Test metrics ({task}):")
                    print(test_metrics)

                if is_regression:
                    predictions_to_write = np.squeeze(test_predictions["logits"])
                else:
                    predictions_to_write = np.argmax(test_predictions["logits"], axis=1)

                output_predict_file = os.path.join(training_args.output_dir, f"predict_results_{task}.txt")
                with open(output_predict_file, "w") as writer:
                    logger.info(f"***** Writing prediction results for {task} *****")
                    writer.write("index\tprediction\n")
                    for index, item in enumerate(predictions_to_write):
                        if is_regression:
                            writer.write(f"{index}\t{item:3.3f}\n")
                        else:
                            item = model.config.id2label[item]
                            writer.write(f"{index}\t{item}\n")
        # endregion

        if training_args.output_dir is not None and not training_args.push_to_hub:
            # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
This code provides an example of how to train a language model using the Transformers library and the Hugging Face framework. The script trains the model on a dataset, performs evaluation and prediction tasks, and returns results.

Here is a high-level summary of what the code does:

* Loads dataset(s)
* Preprocesses data using standard methods
* Sets up training loop with various options (e.g., batch size, epochs, learning rate)
* Trains model for specified number of epochs
* Evaluates model on validation set to compute evaluation metrics
* Optionally uses advanced evaluation settings if MNLI double evaluation is required
* Saves prediction results to output directory

Some notable aspects of the code include:

* The use of Transformers library and Hugging Face framework for training and deployment.
* Support for multiple dataset formats (raw, tf, datasets).
* Custom evaluation metrics (e.g., EM, precision) can be used with additional logging statements.
* Option to save model to local directory when not using hub.


key: chroma:document
string_value:
CODE:

def main():
    # region Argument parsing
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_translation", model_args, data_args, framework="tensorflow")
    # endregion

    # region Logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO)
    datasets.utils.logging.set_verbosity(logging.INFO)
    transformers.utils.logging.set_verbosity(logging.INFO)

    # Log on each process the small summary:
    logger.info(f"Training/evaluation parameters {training_args}")
    # endregion

    # region Detecting last checkpoint
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )
    # endregion

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # region Load datasets
    # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files this script will use the first column for the full texts and the second column for the
    # summaries (unless you specify column names for this with the `text_column` and `summary_column` arguments).
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        raw_datasets = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        raw_datasets = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
        )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading
    # endregion

    # region Load model config and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.

    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast_tokenizer,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )

    prefix = data_args.source_prefix if data_args.source_prefix is not None else ""
    # endregion

    # region Dataset preprocessing
    # We need to tokenize inputs and targets.
    if training_args.do_train:
        column_names = raw_datasets["train"].column_names
    elif training_args.do_eval:
        column_names = raw_datasets["validation"].column_names
    else:
        logger.info("There is nothing to do. Please pass `do_train`, and/or `do_eval`.")
        return

    column_names = raw_datasets["train"].column_names

    # For translation we set the codes of our source and target languages (only useful for mBART, the others will
    # ignore those attributes).
    if isinstance(tokenizer, tuple(MULTILINGUAL_TOKENIZERS)):
        assert data_args.target_lang is not None and data_args.source_lang is not None, (
            f"{tokenizer.__class__.__name__} is a multilingual tokenizer which requires --source_lang and "
            "--target_lang arguments."
        )
        tokenizer.src_lang = data_args.source_lang
        tokenizer.tgt_lang = data_args.target_lang
        forced_bos_token_id = (
            tokenizer.lang_code_to_id[data_args.forced_bos_token] if data_args.forced_bos_token is not None else None
        )

    # Get the language codes for input/target.
    source_lang = data_args.source_lang.split("_")[0]
    target_lang = data_args.target_lang.split("_")[0]

    padding = "max_length" if data_args.pad_to_max_length else False

    # Temporarily set max_target_length for training.
    max_target_length = data_args.max_target_length
    padding = "max_length" if data_args.pad_to_max_length else False

    def preprocess_function(examples):
        inputs = [ex[source_lang] for ex in examples["translation"]]
        targets = [ex[target_lang] for ex in examples["translation"]]
        inputs = [prefix + inp for inp in inputs]
        model_inputs = tokenizer(inputs, max_length=data_args.max_source_length, padding=padding, truncation=True)

        # Tokenize targets with the `text_target` keyword argument
        labels = tokenizer(text_target=targets, max_length=max_target_length, padding=padding, truncation=True)

        # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
        # padding in the loss.
        if padding == "max_length" and data_args.ignore_pad_token_for_loss:
            labels["input_ids"] = [
                [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
            ]

        model_inputs["labels"] = labels["input_ids"]
        return model_inputs

    if training_args.do_train:
        if "train" not in raw_datasets:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = raw_datasets["train"]
        if data_args.max_train_samples is not None:
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))
        train_dataset = train_dataset.map(
            preprocess_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on train dataset",
        )
    else:
        train_dataset = None

    if training_args.do_eval:
        max_target_length = data_args.val_max_target_length
        if "validation" not in raw_datasets:
            raise ValueError("--do_eval requires a validation dataset")
        eval_dataset = raw_datasets["validation"]
        if data_args.max_eval_samples is not None:
            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
            eval_dataset = eval_dataset.select(range(max_eval_samples))
        eval_dataset = eval_dataset.map(
            preprocess_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on validation dataset",
        )
    else:
        eval_dataset = None
    # endregion

    with training_args.strategy.scope():
        # region Prepare model
        model = TFAutoModelForSeq2SeqLM.from_pretrained(
            model_args.model_name_or_path,
            config=config,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )

        # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch
        # on a small vocab and want a smaller embedding size, remove this test.
        embeddings = model.get_input_embeddings()

        # Matt: This is a temporary workaround as we transition our models to exclusively using Keras embeddings.
        #       As soon as the transition is complete, all embeddings should be keras.Embeddings layers, and
        #       the weights will always be in embeddings.embeddings.
        if hasattr(embeddings, "embeddings"):
            embedding_size = embeddings.embeddings.shape[0]
        else:
            embedding_size = embeddings.weight.shape[0]
        if len(tokenizer) > embedding_size:
            model.resize_token_embeddings(len(tokenizer))

        if isinstance(tokenizer, tuple(MULTILINGUAL_TOKENIZERS)):
            model.config.forced_bos_token_id = forced_bos_token_id
        # endregion

        # region Set decoder_start_token_id
        if model.config.decoder_start_token_id is None and isinstance(tokenizer, (MBartTokenizer, MBartTokenizerFast)):
            assert (
                data_args.target_lang is not None and data_args.source_lang is not None
            ), "mBart requires --target_lang and --source_lang"
            if isinstance(tokenizer, MBartTokenizer):
                model.config.decoder_start_token_id = tokenizer.lang_code_to_id[data_args.target_lang]
            else:
                model.config.decoder_start_token_id = tokenizer.convert_tokens_to_ids(data_args.target_lang)

        if model.config.decoder_start_token_id is None:
            raise ValueError("Make sure that `config.decoder_start_token_id` is correctly defined")
        # endregion

        # region Prepare TF Dataset objects
        label_pad_token_id = -100 if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id
        data_collator = DataCollatorForSeq2Seq(
            tokenizer,
            model=model,
            label_pad_token_id=label_pad_token_id,
            pad_to_multiple_of=64,  # Reduce the number of unique shapes for XLA, especially for generation
            return_tensors="np",
        )
        num_replicas = training_args.strategy.num_replicas_in_sync
        total_train_batch_size = training_args.per_device_train_batch_size * num_replicas
        total_eval_batch_size = training_args.per_device_eval_batch_size * num_replicas

        dataset_options = tf.data.Options()
        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF

        # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
        # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
        # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
        # yourself if you use this method, whereas they are automatically inferred from the model input names when
        # using model.prepare_tf_dataset()
        # For more info see the docs:
        # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
        # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset

        tf_train_dataset = model.prepare_tf_dataset(
            train_dataset,
            collate_fn=data_collator,
            batch_size=total_train_batch_size,
            shuffle=True,
        ).with_options(dataset_options)
        tf_eval_dataset = model.prepare_tf_dataset(
            eval_dataset, collate_fn=data_collator, batch_size=total_eval_batch_size, shuffle=False
        ).with_options(dataset_options)
        # endregion

        # region Optimizer and LR scheduling
        num_train_steps = int(len(tf_train_dataset) * training_args.num_train_epochs)
        if training_args.warmup_steps > 0:
            num_warmup_steps = training_args.warmup_steps
        elif training_args.warmup_ratio > 0:
            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
        else:
            num_warmup_steps = 0
        if training_args.do_train:
            optimizer, lr_schedule = create_optimizer(
                init_lr=training_args.learning_rate,
                num_train_steps=num_train_steps,
                num_warmup_steps=num_warmup_steps,
                adam_beta1=training_args.adam_beta1,
                adam_beta2=training_args.adam_beta2,
                adam_epsilon=training_args.adam_epsilon,
                weight_decay_rate=training_args.weight_decay,
                adam_global_clipnorm=training_args.max_grad_norm,
            )
        else:
            optimizer = "sgd"  # Just write anything because we won't be using it
        # endregion

        # region Metric and postprocessing
        if training_args.do_eval:
            metric = evaluate.load("sacrebleu", cache_dir=model_args.cache_dir)

            if data_args.val_max_target_length is None:
                data_args.val_max_target_length = data_args.max_target_length

            gen_kwargs = {
                "max_length": data_args.val_max_target_length,
                "num_beams": data_args.num_beams,
                "no_repeat_ngram_size": 0,  # Not supported under XLA right now, and some models set it by default
            }

            def postprocess_text(preds, labels):
                preds = [pred.strip() for pred in preds]
                labels = [[label.strip()] for label in labels]

                return preds, labels

            def compute_metrics(preds):
                predictions, labels = preds
                if isinstance(predictions, tuple):
                    predictions = predictions[0]
                decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
                labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
                decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
                decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)
                metrics = metric.compute(predictions=decoded_preds, references=decoded_labels)
                return {"bleu": metrics["score"]}

            # The KerasMetricCallback allows metrics that are too complex to write as standard Keras metrics
            # to be computed each epoch. Any Python code can be included in the metric_fn. This is especially
            # useful for metrics like BLEU and ROUGE that perform string comparisons on decoded model outputs.
            # For more information, see the docs at
            # https://huggingface.co/docs/transformers/main_classes/keras_callbacks#transformers.KerasMetricCallback

            metric_callback = KerasMetricCallback(
                metric_fn=compute_metrics,
                eval_dataset=tf_eval_dataset,
                predict_with_generate=True,
                use_xla_generation=True,
                generate_kwargs=gen_kwargs,
            )
            callbacks = [metric_callback]
        else:
            callbacks = []

        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            push_to_hub_model_id = f"{model_name}-finetuned-{data_args.source_lang}-{data_args.target_lang}"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "translation"}
        if data_args.dataset_name is not None:
            model_card_kwargs["dataset_tags"] = data_args.dataset_name
            if data_args.dataset_config_name is not None:
                model_card_kwargs["dataset_args"] = data_args.dataset_config_name
                model_card_kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
            else:
                model_card_kwargs["dataset"] = data_args.dataset_name

        languages = [l for l in [data_args.source_lang, data_args.target_lang] if l is not None]
        if len(languages) > 0:
            model_card_kwargs["language"] = languages

        if training_args.push_to_hub:
            # Because this training can be quite long, we save once per epoch.
            callbacks.append(
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            )
        # endregion

        # region Training
        eval_metrics = None
        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, jit_compile=training_args.xla)

        if training_args.do_train:
            logger.info("***** Running training *****")
            logger.info(f"  Num examples = {len(train_dataset)}")
            logger.info(f"  Num Epochs = {training_args.num_train_epochs}")
            logger.info(f"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}")
            logger.info(f"  Total train batch size = {total_train_batch_size}")
            logger.info(f"  Total optimization steps = {num_train_steps}")

            if training_args.xla and not data_args.pad_to_max_length:
                logger.warning(
                    "XLA training may be slow at first when --pad_to_max_length is not set "
                    "until all possible shapes have been compiled."
                )

            history = model.fit(tf_train_dataset, epochs=int(training_args.num_train_epochs), callbacks=callbacks)
            eval_metrics = {key: val[-1] for key, val in history.history.items()}
        # endregion

        # region Validation
        if training_args.do_eval and not training_args.do_train:
            # Compiling generation with XLA yields enormous speedups, see https://huggingface.co/blog/tf-xla-generate
            @tf.function(jit_compile=True)
            def generate(**kwargs):
                return model.generate(**kwargs)

            if training_args.do_eval:
                logger.info("Evaluation...")
                for batch, labels in tf_eval_dataset:
                    batch.update(gen_kwargs)
                    generated_tokens = generate(**batch)
                    if isinstance(generated_tokens, tuple):
                        generated_tokens = generated_tokens[0]
                    decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
                    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
                    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
                    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

                    metric.add_batch(predictions=decoded_preds, references=decoded_labels)

                eval_metrics = metric.compute()
                logger.info({"bleu": eval_metrics["score"]})
        # endregion

        if training_args.output_dir is not None and eval_metrics is not None:
            output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
            with open(output_eval_file, "w") as writer:
                writer.write(json.dumps(eval_metrics))

        if training_args.output_dir is not None and not training_args.push_to_hub:
            # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
This code is for training a machine translation model using the Hugging Face Transformers library. The script trains the model on a specified dataset and evaluates its performance by calculating the BLEU score, saving results to a file, and potentially uploading the trained model to Hub.

The main components of the script include:

1. Setting up the model and tokenizer
2. Preparing training data
3. Defining evaluation metrics
4. Compiling the model
5. Training the model
6. Evaluating the model

The code also includes various callbacks that are used during training, such as `PushToHubCallback` to upload the trained model to Hub, and `KerasMetricCallback` to evaluate the model's performance on a separate dataset.

Here is a concise summary of the script:

* Set up machine translation model and tokenizer
* Prepare training data
* Define evaluation metrics (BLEU score)
* Compile the model
* Train the model
* Evaluate the model using BLEU score

Overall, this code provides a robust framework for training machine translation models on large datasets using popular libraries like Hugging Face Transformers.


key: chroma:document
string_value:
CODE:

def main():
    # region Argument parsing
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_swag", model_args, data_args, framework="tensorflow")

    output_dir = Path(training_args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)
    # endregion

    # region Logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    log_level = training_args.get_process_log_level()
    logger.setLevel(log_level)
    datasets.utils.logging.set_verbosity(log_level)
    transformers.utils.logging.set_verbosity(log_level)
    transformers.utils.logging.enable_default_handler()
    transformers.utils.logging.enable_explicit_format()
    # endregion

    # region Checkpoints
    checkpoint = None
    if len(os.listdir(training_args.output_dir)) > 0 and not training_args.overwrite_output_dir:
        if (output_dir / CONFIG_NAME).is_file() and (output_dir / TF2_WEIGHTS_NAME).is_file():
            checkpoint = output_dir
            logger.info(
                f"Checkpoint detected, resuming training from checkpoint in {training_args.output_dir}. To avoid this"
                " behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )
        else:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to continue regardless."
            )
    # endregion

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # region Load datasets
    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).

    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).

    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.train_file is not None or data_args.validation_file is not None:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        raw_datasets = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
        )
    else:
        # Downloading and loading the swag dataset from the hub.
        raw_datasets = load_dataset(
            "swag",
            "regular",
            cache_dir=model_args.cache_dir,
            token=model_args.token,
        )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    # When using your own dataset or a different dataset from swag, you will probably need to change this.
    ending_names = [f"ending{i}" for i in range(4)]
    context_name = "sent1"
    question_header_name = "sent2"
    # endregion

    # region Load model config and tokenizer
    if checkpoint is not None:
        config_path = training_args.output_dir
    elif model_args.config_name:
        config_path = model_args.config_name
    else:
        config_path = model_args.model_name_or_path

    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    config = AutoConfig.from_pretrained(
        config_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast_tokenizer,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    # endregion

    # region Dataset preprocessing
    if data_args.max_seq_length is None:
        max_seq_length = tokenizer.model_max_length
        if max_seq_length > 1024:
            logger.warning(
                f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). "
                "Picking 1024 instead. You can change that default value by passing --max_seq_length xxx."
            )
            max_seq_length = 1024
    else:
        if data_args.max_seq_length > tokenizer.model_max_length:
            logger.warning(
                f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the "
                f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
            )
        max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    def preprocess_function(examples):
        first_sentences = [[context] * 4 for context in examples[context_name]]
        question_headers = examples[question_header_name]
        second_sentences = [
            [f"{header} {examples[end][i]}" for end in ending_names] for i, header in enumerate(question_headers)
        ]

        # Flatten out
        first_sentences = list(chain(*first_sentences))
        second_sentences = list(chain(*second_sentences))

        # Tokenize
        tokenized_examples = tokenizer(first_sentences, second_sentences, truncation=True, max_length=max_seq_length)
        # Un-flatten
        data = {k: [v[i : i + 4] for i in range(0, len(v), 4)] for k, v in tokenized_examples.items()}
        return data

    if training_args.do_train:
        if "train" not in raw_datasets:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = raw_datasets["train"]
        if data_args.max_train_samples is not None:
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))
        train_dataset = train_dataset.map(
            preprocess_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    if training_args.do_eval:
        if "validation" not in raw_datasets:
            raise ValueError("--do_eval requires a validation dataset")
        eval_dataset = raw_datasets["validation"]
        if data_args.max_eval_samples is not None:
            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
            eval_dataset = eval_dataset.select(range(max_eval_samples))
        eval_dataset = eval_dataset.map(
            preprocess_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    if data_args.pad_to_max_length:
        data_collator = DefaultDataCollator(return_tensors="np")
    else:
        # custom class defined above, as HF has no data collator for multiple choice
        data_collator = DataCollatorForMultipleChoice(tokenizer)
    # endregion

    with training_args.strategy.scope():
        # region Build model
        if checkpoint is None:
            model_path = model_args.model_name_or_path
        else:
            model_path = checkpoint
        model = TFAutoModelForMultipleChoice.from_pretrained(
            model_path,
            config=config,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )

        num_replicas = training_args.strategy.num_replicas_in_sync
        total_train_batch_size = training_args.per_device_train_batch_size * num_replicas
        total_eval_batch_size = training_args.per_device_eval_batch_size * num_replicas

        if training_args.do_train:
            num_train_steps = (len(train_dataset) // total_train_batch_size) * int(training_args.num_train_epochs)
            if training_args.warmup_steps > 0:
                num_warmup_steps = training_args.warmup_steps
            elif training_args.warmup_ratio > 0:
                num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
            else:
                num_warmup_steps = 0
            optimizer, lr_schedule = create_optimizer(
                init_lr=training_args.learning_rate,
                num_train_steps=num_train_steps,
                num_warmup_steps=num_warmup_steps,
                adam_beta1=training_args.adam_beta1,
                adam_beta2=training_args.adam_beta2,
                adam_epsilon=training_args.adam_epsilon,
                weight_decay_rate=training_args.weight_decay,
                adam_global_clipnorm=training_args.max_grad_norm,
            )
        else:
            optimizer = "sgd"  # Just write anything because we won't be using it
        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, metrics=["accuracy"], jit_compile=training_args.xla)
        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            push_to_hub_model_id = f"{model_name}-finetuned-multiplechoice"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "multiple-choice"}

        if training_args.push_to_hub:
            callbacks = [
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            ]
        else:
            callbacks = []
        # endregion

        # region Training
        eval_metrics = None
        if training_args.do_train:
            dataset_options = tf.data.Options()
            dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF

            # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
            # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
            # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
            # yourself if you use this method, whereas they are automatically inferred from the model input names when
            # using model.prepare_tf_dataset()
            # For more info see the docs:
            # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
            # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset

            tf_train_dataset = model.prepare_tf_dataset(
                train_dataset,
                shuffle=True,
                batch_size=total_train_batch_size,
                collate_fn=data_collator,
            ).with_options(dataset_options)

            if training_args.do_eval:
                validation_data = model.prepare_tf_dataset(
                    eval_dataset,
                    shuffle=False,
                    batch_size=total_eval_batch_size,
                    collate_fn=data_collator,
                    drop_remainder=True,
                ).with_options(dataset_options)
            else:
                validation_data = None
            history = model.fit(
                tf_train_dataset,
                validation_data=validation_data,
                epochs=int(training_args.num_train_epochs),
                callbacks=callbacks,
            )
            eval_metrics = {key: val[-1] for key, val in history.history.items()}
        # endregion

        # region Evaluation
        if training_args.do_eval and not training_args.do_train:
            dataset_options = tf.data.Options()
            dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF
            # Do a standalone evaluation pass
            tf_eval_dataset = model.prepare_tf_dataset(
                eval_dataset,
                shuffle=False,
                batch_size=total_eval_batch_size,
                collate_fn=data_collator,
                drop_remainder=True,
            ).with_options(dataset_options)
            eval_results = model.evaluate(tf_eval_dataset)
            eval_metrics = {"val_loss": eval_results[0], "val_accuracy": eval_results[1]}
        # endregion

        if eval_metrics is not None and training_args.output_dir is not None:
            output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
            with open(output_eval_file, "w") as writer:
                writer.write(json.dumps(eval_metrics))

        # region Push to hub

        if training_args.output_dir is not None and not training_args.push_to_hub:
            # If we're not pushing to hub, at least save a local copy when we're done
            model.save_pretrained(training_args.output_dir)


SUMMARY:
This is a code snippet for training and evaluating a multiple-choice conversational AI model using the Transformers library by Hugging Face. The code defines various parameters, loads pre-trained models, fine-tunes them on a specific task, trains and evaluates the models, saves the results, and optional pushes the updated weights to a Hub.

Here's a concise summary of the main parts:

1. **Model Preparation**:
   - Defines parameters for training and evaluation.
   - Prepares a Hugging Face dataset for training and evaluation.
   - Initializes pre-trained models and fine-tuning task specifics.

2. **Training Loop**:
   - Creates an optimizer, scheduler, and loss function.
   - Trains the model on the dataset using the training method from Transformers's TFEf Trainer class.

3. **Evaluation**:
   - Evaluates the trained model on the validation dataset if training took place.

4. **Results Saving**:
   - Saves the evaluation metrics or results in a JSON file named according to the directory where `output_dir` was specified.

5. **Pushing to Hub (optional)**:
   - If `push_to_hub` is True and `output_dir` exists, pushes the updated weights of the model to the chosen hub in Hugging Face.

Note that without knowing the exact implementation details and context of this specific code snippet, some steps or parts might be simplified or described as suggested by the authors.


