docstr: Args:
      input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
          Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you
          provide it.
  
          Indices can be obtained using [`Speech2Text2Tokenizer`]. See [`PreTrainedTokenizer.encode`] and
          [`PreTrainedTokenizer.__call__`] for details.
  
          [What are input IDs?](../glossary#input-ids)
      attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
          Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  
          - 1 for tokens that are **not masked**,
          - 0 for tokens that are **masked**.
  
          [What are attention masks?](../glossary#attention-mask)
      encoder_hidden_states  (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
          Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention
          if the model is configured as a decoder.
      encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*):
          Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used
          in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
      head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):
          Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`:
  
          - 1 indicates the head is **not masked**,
          - 0 indicates the head is **masked**.
  
      cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):
          Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`:
  
          - 1 indicates the head is **not masked**,
          - 0 indicates the head is **masked**.
  
      past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
          Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of
          shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of
          shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. The two additional
          tensors are only required when the model is used as a decoder in a Sequence to Sequence model.
  
          Contains pre-computed hidden-states (key and values in the self-attention blocks and in the
          cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.
  
          If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those
          that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of
          all `decoder_input_ids` of shape `(batch_size, sequence_length)`.
      labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
          Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
          config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
          (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
      use_cache (`bool`, *optional*):
          If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
          (see `past_key_values`).
  
          - 1 for tokens that are **not masked**,
          - 0 for tokens that are **masked**.
      output_attentions (`bool`, *optional*):
          Whether or not to return the attentions tensors of all attention layers. See `attentions` under
          returned tensors for more detail.
      output_hidden_states (`bool`, *optional*):
          Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
          for more detail.
      return_dict (`bool`, *optional*):
          Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
  
  Returns:
  
  Example:
  
  ```python
  >>> from transformers import (
  ...     SpeechEncoderDecoderModel,
  ...     Speech2Text2ForCausalLM,
  ...     Wav2Vec2Model,
  ...     Speech2Text2Config,
  ...     Wav2Vec2Config,
  ...     Wav2Vec2FeatureExtractor,
  ...     Speech2Text2Tokenizer,
  ... )
  >>> from datasets import load_dataset
  
  >>> feature_extractor = Wav2Vec2FeatureExtractor()
  >>> tokenizer = Speech2Text2Tokenizer.from_pretrained("facebook/s2t-wav2vec2-large-en-de")
  
  >>> encoder = Wav2Vec2Model(Wav2Vec2Config())
  >>> decoder = Speech2Text2ForCausalLM(Speech2Text2Config())
  >>> # init random speech2text model
  
  >>> model = SpeechEncoderDecoderModel(encoder=encoder, decoder=decoder)
  >>> model.config.pad_token_id = tokenizer.pad_token_id
  >>> model.config.decoder_start_token_id = tokenizer.bos_token_id
  >>> # pre-process inputs and labels
  
  >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
  >>> inputs = feature_extractor(
  ...     ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt"
  ... )
  >>> input_values = inputs.input_values
  >>> decoder_input_ids = tokenizer(ds[0]["text"], return_tensors="pt").input_ids
  >>> # compute loss
  
  >>> loss = model(inputs=input_values, labels=decoder_input_ids).loss
  >>> # backprop loss
  
  >>> loss.backward()  # doctest: +IGNORE_RESULT
  ```
docstr_tok: ["Args",":","input_ids","(","`","torch.LongTensor","`","of","shape","`","(","batch_size",",","sequence_length",")","`",")",":","Indices","of","input","sequence","tokens","in","the","vocabulary",".","Padding","will","be","ignored","by","default","should","you","provide","it",".","Indices","can","be","obtained","using","[","`","Speech2Text2Tokenizer","`","]",".","See","[","`","PreTrainedTokenizer.encode","`","]","and","[","`","PreTrainedTokenizer.__call__","`","]","for","details",".","[","What","are","input","IDs","?","]","(","..","\/glossary","#","input-ids",")","attention_mask","(","`","torch.Tensor","`","of","shape","`","(","batch_size",",","sequence_length",")","`",",","*","optional","*",")",":","Mask","to","avoid","performing","attention","on","padding","token","indices",".","Mask","values","selected","in","`","[","0",",","1","]","`",":","-","1","for","tokens","that","are","*","*","not","masked","*","*",",","-","0","for","tokens","that","are","*","*","masked","*","*",".","[","What","are","attention","masks","?","]","(","..","\/glossary","#","attention-mask",")","encoder_hidden_states","(","`","torch.FloatTensor","`","of","shape","`","(","batch_size",",","sequence_length",",","hidden_size",")","`",",","*","optional","*",")",":","Sequence","of","hidden-states","at","the","output","of","the","last","layer","of","the","encoder",".","Used","in","the","cross-attention","if","the","model","is","configured","as","a","decoder",".","encoder_attention_mask","(","`","torch.FloatTensor","`","of","shape","`","(","batch_size",",","sequence_length",")","`",",","*","optional","*",")",":","Mask","to","avoid","performing","attention","on","the","padding","token","indices","of","the","encoder","input",".","This","mask","is","used","in","the","cross-attention","if","the","model","is","configured","as","a","decoder",".","Mask","values","selected","in","`","[","0",",","1","]","`",":","head_mask","(","`","torch.Tensor","`","of","shape","`","(","decoder_layers",",","decoder_attention_heads",")","`",",","*","optional","*",")",":","Mask","to","nullify","selected","heads","of","the","attention","modules",".","Mask","values","selected","in","`","[","0",",","1","]","`",":","-","1","indicates","the","head","is","*","*","not","masked","*","*",",","-","0","indicates","the","head","is","*","*","masked","*","*",".","cross_attn_head_mask","(","`","torch.Tensor","`","of","shape","`","(","decoder_layers",",","decoder_attention_heads",")","`",",","*","optional","*",")",":","Mask","to","nullify","selected","heads","of","the","cross-attention","modules",".","Mask","values","selected","in","`","[","0",",","1","]","`",":","-","1","indicates","the","head","is","*","*","not","masked","*","*",",","-","0","indicates","the","head","is","*","*","masked","*","*",".","past_key_values","(","`","tuple","(","tuple","(","torch.FloatTensor",")",")","`",",","*","optional","*",",","returned","when","`","use_cache=True","`","is","passed","or","when","`","config.use_cache=True","`",")",":","Tuple","of","`","tuple","(","torch.FloatTensor",")","`","of","length","`","config.n_layers","`",",","with","each","tuple","having","2","tensors","of","shape","`","(","batch_size",",","num_heads",",","sequence_length",",","embed_size_per_head",")","`",")","and","2","additional","tensors","of","shape","`","(","batch_size",",","num_heads",",","encoder_sequence_length",",","embed_size_per_head",")","`",".","The","two","additional","tensors","are","only","required","when","the","model","is","used","as","a","decoder","in","a","Sequence","to","Sequence","model",".","Contains","pre-computed","hidden-states","(","key","and","values","in","the","self-attention","blocks","and","in","the","cross-attention","blocks",")","that","can","be","used","(","see","`","past_key_values","`","input",")","to","speed","up","sequential","decoding",".","If","`","past_key_values","`","are","used",",","the","user","can","optionally","input","only","the","last","`","decoder_input_ids","`","(","those","that","do","n't","have","their","past","key","value","states","given","to","this","model",")","of","shape","`","(","batch_size",",","1",")","`","instead","of","all","`","decoder_input_ids","`","of","shape","`","(","batch_size",",","sequence_length",")","`",".","labels","(","`","torch.LongTensor","`","of","shape","`","(","batch_size",",","sequence_length",")","`",",","*","optional","*",")",":","Labels","for","computing","the","masked","language","modeling","loss",".","Indices","should","either","be","in","`","[","0",",","...",",","config.vocab_size","]","`","or","-100","(","see","`","input_ids","`","docstring",")",".","Tokens","with","indices","set","to","`","-100","`","are","ignored","(","masked",")",",","the","loss","is","only","computed","for","the","tokens","with","labels","in","`","[","0",",","...",",","config.vocab_size","]","`",".","use_cache","(","`","bool","`",",","*","optional","*",")",":","If","set","to","`","True","`",",","`","past_key_values","`","key","value","states","are","returned","and","can","be","used","to","speed","up","decoding","(","see","`","past_key_values","`",")",".","-","1","for","tokens","that","are","*","*","not","masked","*","*",",","-","0","for","tokens","that","are","*","*","masked","*","*",".","output_attentions","(","`","bool","`",",","*","optional","*",")",":","Whether","or","not","to","return","the","attentions","tensors","of","all","attention","layers",".","See","`","attentions","`","under","returned","tensors","for","more","detail",".","output_hidden_states","(","`","bool","`",",","*","optional","*",")",":","Whether","or","not","to","return","the","hidden","states","of","all","layers",".","See","`","hidden_states","`","under","returned","tensors","for","more","detail",".","return_dict","(","`","bool","`",",","*","optional","*",")",":","Whether","or","not","to","return","a","[","`","~utils.ModelOutput","`","]","instead","of","a","plain","tuple",".","Returns",":","Example",":","``","`","python",">",">",">","from","transformers","import","(","...","SpeechEncoderDecoderModel",",","...","Speech2Text2ForCausalLM",",","...","Wav2Vec2Model",",","...","Speech2Text2Config",",","...","Wav2Vec2Config",",","...","Wav2Vec2FeatureExtractor",",","...","Speech2Text2Tokenizer",",","...",")",">",">",">","from","datasets","import","load_dataset",">",">",">","feature_extractor","=","Wav2Vec2FeatureExtractor","(",")",">",">",">","tokenizer","=","Speech2Text2Tokenizer.from_pretrained","(","``","facebook\/s2t-wav2vec2-large-en-de","''",")",">",">",">","encoder","=","Wav2Vec2Model","(","Wav2Vec2Config","(",")",")",">",">",">","decoder","=","Speech2Text2ForCausalLM","(","Speech2Text2Config","(",")",")",">",">",">","#","init","random","speech2text","model",">",">",">","model","=","SpeechEncoderDecoderModel","(","encoder=encoder",",","decoder=decoder",")",">",">",">","model.config.pad_token_id","=","tokenizer.pad_token_id",">",">",">","model.config.decoder_start_token_id","=","tokenizer.bos_token_id",">",">",">","#","pre-process","inputs","and","labels",">",">",">","ds","=","load_dataset","(","``","hf-internal-testing\/librispeech_asr_dummy","''",",","``","clean","''",",","split=","''","validation","''",")",">",">",">","inputs","=","feature_extractor","(","...","ds","[","0","]","[","``","audio","''","]","[","``","array","''","]",",","sampling_rate=ds","[","0","]","[","``","audio","''","]","[","``","sampling_rate","''","]",",","return_tensors=","''","pt","''","...",")",">",">",">","input_values","=","inputs.input_values",">",">",">","decoder_input_ids","=","tokenizer","(","ds","[","0","]","[","``","text","''","]",",","return_tensors=","''","pt","''",")",".input_ids",">",">",">","#","compute","loss",">",">",">","loss","=","model","(","inputs=input_values",",","labels=decoder_input_ids",")",".loss",">",">",">","#","backprop","loss",">",">",">","loss.backward","(",")","#","doctest",":","+IGNORE_RESULT","``","`"]
code: def forward(
      self,
      input_ids: Optional[torch.LongTensor] = None,
      attention_mask: Optional[torch.Tensor] = None,
      encoder_hidden_states: Optional[torch.FloatTensor] = None,
      encoder_attention_mask: Optional[torch.FloatTensor] = None,
      head_mask: Optional[torch.Tensor] = None,
      cross_attn_head_mask: Optional[torch.Tensor] = None,
      past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
      inputs_embeds: Optional[torch.FloatTensor] = None,
      labels: Optional[torch.LongTensor] = None,
      use_cache: Optional[bool] = None,
      output_attentions: Optional[bool] = None,
      output_hidden_states: Optional[bool] = None,
      return_dict: Optional[bool] = None,
  ) -> Union[
      Tuple[torch.FloatTensor], CausalLMOutputWithCrossAttentions
  ]:
      r"""
      Args:
          input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
              Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you
              provide it.
  
              Indices can be obtained using [`Speech2Text2Tokenizer`]. See [`PreTrainedTokenizer.encode`] and
              [`PreTrainedTokenizer.__call__`] for details.
  
              [What are input IDs?](../glossary#input-ids)
          attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
              Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  
              - 1 for tokens that are **not masked**,
              - 0 for tokens that are **masked**.
  
              [What are attention masks?](../glossary#attention-mask)
          encoder_hidden_states  (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
              Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention
              if the model is configured as a decoder.
          encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*):
              Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used
              in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
          head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):
              Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`:
  
              - 1 indicates the head is **not masked**,
              - 0 indicates the head is **masked**.
  
          cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):
              Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`:
  
              - 1 indicates the head is **not masked**,
              - 0 indicates the head is **masked**.
  
          past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
              Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of
              shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of
              shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. The two additional
              tensors are only required when the model is used as a decoder in a Sequence to Sequence model.
  
              Contains pre-computed hidden-states (key and values in the self-attention blocks and in the
              cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.
  
              If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those
              that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of
              all `decoder_input_ids` of shape `(batch_size, sequence_length)`.
          labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
              Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
              config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
              (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
          use_cache (`bool`, *optional*):
              If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
              (see `past_key_values`).
  
              - 1 for tokens that are **not masked**,
              - 0 for tokens that are **masked**.
          output_attentions (`bool`, *optional*):
              Whether or not to return the attentions tensors of all attention layers. See `attentions` under
              returned tensors for more detail.
          output_hidden_states (`bool`, *optional*):
              Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
              for more detail.
          return_dict (`bool`, *optional*):
              Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
  
      Returns:
  
      Example:
  
      ```python
      >>> from transformers import (
      ...     SpeechEncoderDecoderModel,
      ...     Speech2Text2ForCausalLM,
      ...     Wav2Vec2Model,
      ...     Speech2Text2Config,
      ...     Wav2Vec2Config,
      ...     Wav2Vec2FeatureExtractor,
      ...     Speech2Text2Tokenizer,
      ... )
      >>> from datasets import load_dataset
  
      >>> feature_extractor = Wav2Vec2FeatureExtractor()
      >>> tokenizer = Speech2Text2Tokenizer.from_pretrained("facebook/s2t-wav2vec2-large-en-de")
  
      >>> encoder = Wav2Vec2Model(Wav2Vec2Config())
      >>> decoder = Speech2Text2ForCausalLM(Speech2Text2Config())
      >>> # init random speech2text model
  
      >>> model = SpeechEncoderDecoderModel(encoder=encoder, decoder=decoder)
      >>> model.config.pad_token_id = tokenizer.pad_token_id
      >>> model.config.decoder_start_token_id = tokenizer.bos_token_id
      >>> # pre-process inputs and labels
  
      >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
      >>> inputs = feature_extractor(
      ...     ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt"
      ... )
      >>> input_values = inputs.input_values
      >>> decoder_input_ids = tokenizer(ds[0]["text"], return_tensors="pt").input_ids
      >>> # compute loss
  
      >>> loss = model(inputs=input_values, labels=decoder_input_ids).loss
      >>> # backprop loss
  
      >>> loss.backward()  # doctest: +IGNORE_RESULT
      ```"""
  
      output_attentions = (
          output_attentions
          if output_attentions is not None
          else self.config.output_attentions
      )
      output_hidden_states = (
          output_hidden_states
          if output_hidden_states is not None
          else self.config.output_hidden_states
      )
      return_dict = (
          return_dict
          if return_dict is not None
          else self.config.use_return_dict
      )
  
      # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
      outputs = self.model.decoder(
          input_ids=input_ids,
          attention_mask=attention_mask,
          encoder_hidden_states=encoder_hidden_states,
          encoder_attention_mask=encoder_attention_mask,
          head_mask=head_mask,
          cross_attn_head_mask=cross_attn_head_mask,
          past_key_values=past_key_values,
          inputs_embeds=inputs_embeds,
          use_cache=use_cache,
          output_attentions=output_attentions,
          output_hidden_states=output_hidden_states,
          return_dict=return_dict,
      )
  
      logits = self.lm_head(outputs[0])
  
      loss = None
      if labels is not None:
          loss_fct = CrossEntropyLoss()
          loss = loss_fct(
              logits.view(-1, self.config.vocab_size), labels.view(-1)
          )
  
      if not return_dict:
          output = (logits,) + outputs[1:]
          return (loss,) + output if loss is not None else output
  
      return CausalLMOutputWithCrossAttentions(
          loss=loss,
          logits=logits,
          past_key_values=outputs.past_key_values,
          hidden_states=outputs.hidden_states,
          attentions=outputs.attentions,
          cross_attentions=outputs.cross_attentions,
      )
  
imports: import copy
  import math
  from typing import Optional, Tuple, Union
  import torch
  from torch import nn
  from torch.nn import CrossEntropyLoss
  from ...activations import ACT2FN
  from ...modeling_attn_mask_utils import _prepare_4d_attention_mask, _prepare_4d_causal_attention_mask
  from ...modeling_outputs import BaseModelOutputWithPastAndCrossAttentions, CausalLMOutputWithCrossAttentions
  from ...modeling_utils import PreTrainedModel
  from ...utils import add_start_docstrings, logging, replace_return_docstrings
  from .configuration_speech_to_text_2 import Speech2Text2Config
label_desc: Use image node_id 8 for calling the Speech2Text2ForCausalLM obj's underlying member method code with example usage: obj.forward(input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, head_mask, cross_attn_head_mask, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict) and returns: CausalLMOutputWithCrossAttentions, unknown