class_docstr: A utility attention mask class that allows one to:
      - Create a causal 4d mask
      - Create a causal 4d mask with slided window
      - Convert a 2d attention mask (batch_size, query_length) to a 4d attention mask (batch_size, 1, query_length,
        key_value_length) that can be multiplied with attention scores
  
  Examples:
  
  ```python
  >>> import torch
  >>> from transformers.modeling_attn_mask_utils import AttentionMaskConverter
  
  >>> converter = AttentionMaskConverter(True)
  >>> converter.to_4d(torch.tensor([[0, 0, 0, 1, 1]]), 5, key_value_length=5, dtype=torch.float32)
  tensor([[[[-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38],
          [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38],
          [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38],
          [-3.4028e+38, -3.4028e+38, -3.4028e+38,  0.0000e+00, -3.4028e+38],
          [-3.4028e+38, -3.4028e+38, -3.4028e+38,  0.0000e+00,  0.0000e+00]]]])
  ```
  
  Parameters:
      is_causal (`bool`):
          Whether the attention mask should be a uni-directional (causal) or bi-directional mask.
  
      sliding_window (`int`, *optional*):
          Optionally, the sliding window masks can be created if `sliding_window` is defined to a positive integer.
class_docstr_tok: ["A","utility","attention","mask","class","that","allows","one","to",":","-","Create","a","causal","4d","mask","-","Create","a","causal","4d","mask","with","slided","window","-","Convert","a","2d","attention","mask","(","batch_size",",","query_length",")","to","a","4d","attention","mask","(","batch_size",",","1",",","query_length",",","key_value_length",")","that","can","be","multiplied","with","attention","scores","Examples",":","``","`","python",">",">",">","import","torch",">",">",">","from","transformers.modeling_attn_mask_utils","import","AttentionMaskConverter",">",">",">","converter","=","AttentionMaskConverter","(","True",")",">",">",">","converter.to_4d","(","torch.tensor","(","[","[","0",",","0",",","0",",","1",",","1","]","]",")",",","5",",","key_value_length=5",",","dtype=torch.float32",")","tensor","(","[","[","[","[","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38","]",",","[","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38","]",",","[","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38","]",",","[","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38",",","0.0000e+00",",","-3.4028e+38","]",",","[","-3.4028e+38",",","-3.4028e+38",",","-3.4028e+38",",","0.0000e+00",",","0.0000e+00","]","]","]","]",")","``","`","Parameters",":","is_causal","(","`","bool","`",")",":","Whether","the","attention","mask","should","be","a","uni-directional","(","causal",")","or","bi-directional","mask",".","sliding_window","(","`","int","`",",","*","optional","*",")",":","Optionally",",","the","sliding","window","masks","can","be","created","if","`","sliding_window","`","is","defined","to","a","positive","integer","."]
docstr: Converts 2D attention mask to 4D attention mask by expanding mask to (bsz, head_dim=1, query_length,
  key_value_length) shape and by adding a large negative bias to not-attended positions. If attention_mask is
  causal, a causal mask will be added.
docstr_tok: ["Converts","2D","attention","mask","to","4D","attention","mask","by","expanding","mask","to","(","bsz",",","head_dim=1",",","query_length",",","key_value_length",")","shape","and","by","adding","a","large","negative","bias","to","not-attended","positions",".","If","attention_mask","is","causal",",","a","causal","mask","will","be","added","."]
code: def to_4d(
      self,
      attention_mask_2d: torch.Tensor,
      query_length: int,
      dtype: torch.dtype,
      key_value_length: Optional[int] = None,
  ) -> torch.Tensor:
      """
      Converts 2D attention mask to 4D attention mask by expanding mask to (bsz, head_dim=1, query_length,
      key_value_length) shape and by adding a large negative bias to not-attended positions. If attention_mask is
      causal, a causal mask will be added.
      """
      input_shape = (attention_mask_2d.shape[0], query_length)
  
      # create causal mask
      # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
      causal_4d_mask = None
      if (
          input_shape[-1] > 1 or self.sliding_window is not None
      ) and self.is_causal:
          if key_value_length is None:
              raise ValueError(
                  "This attention mask converter is causal. Make sure to pass `key_value_length` to correctly create a causal mask."
              )
  
          past_key_values_length = key_value_length - query_length
          causal_4d_mask = self._make_causal_mask(
              input_shape,
              dtype,
              device=attention_mask_2d.device,
              past_key_values_length=past_key_values_length,
              sliding_window=self.sliding_window,
          )
      elif self.sliding_window is not None:
          raise NotImplementedError(
              "Sliding window is currently only implemented for causal masking"
          )
  
      # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
      expanded_attn_mask = self._expand_mask(
          attention_mask_2d, dtype, tgt_len=input_shape[-1]
      ).to(attention_mask_2d.device)
      if causal_4d_mask is not None:
          expanded_attn_mask = causal_4d_mask.masked_fill(
              expanded_attn_mask.bool(), torch.finfo(dtype).min
          )
  
      # expanded_attn_mask + causal_4d_mask can cause some overflow
      expanded_4d_mask = expanded_attn_mask
  
      return expanded_4d_mask
  
file_path: transformers/src/transformers/modeling_attn_mask_utils.py
imports: from dataclasses import dataclass
  from typing import List, Optional, Tuple, Union
  import torch
label_desc: Use image node_id 3 for calling the AttentionMaskConverter obj's underlying member method code with example usage: obj.to_4d(attention_mask_2d, query_length, dtype, key_value_length) and returns: expanded_4d_mask