class_docstr: Outputs: `Tuple` comprising various elements depending on the configuration
      (config) and inputs:
      **last_hidden_state**: ``torch.FloatTensor`` of shape
          ``(batch_size, sequence_length, hidden_size)``. Sequence of
          hidden-states at the output of the last layer of the model.
      **pooler_output**: ``torch.FloatTensor`` of shape
          ``(batch_size, hidden_size)``. Last layer hidden-state of the
          first token of the sequence (classification token) further processed
          by a Linear layer and a Tanh activation function. The Linear
          layer weights are trained from the next sentence prediction
          (classification) objective during Bert pretraining. This output
          is usually *not* a good summary of the semantic content of the
          input, you're often better with averaging or pooling
          the sequence of hidden-states for the whole input sequence.
      **hidden_states**: (`optional`, returned when
          ``config.output_hidden_states=True``)
          list of ``torch.FloatTensor`` (one for the output of each layer +
          the output of the embeddings)
          of shape ``(batch_size, sequence_length, hidden_size)``:
          Hidden-states of the model at the output of each layer plus the
          initial embedding outputs.
      **attentions**: (`optional`, returned when
          ``config.output_attentions=True``) list of ``torch.FloatTensor``
          (one for each layer) of shape ``(batch_size, num_heads,
          sequence_length, sequence_length)``: Attentions weights after
          the attention softmax, used to compute the weighted average in the
          self-attention heads.
  Examples::
      # For example purposes. Not runnable.
      transformer = BertModel.from_pretrained('bert-base-uncased')
      encoder = ImageEncoder(args)
      mmbt = MMBTModel(config, transformer, encoder)
class_docstr_tok: ["Outputs",":","`","Tuple","`","comprising","various","elements","depending","on","the","configuration","(","config",")","and","inputs",":","*","*","last_hidden_state","*","*",":","``","torch.FloatTensor","``","of","shape","``","(","batch_size",",","sequence_length",",","hidden_size",")","``",".","Sequence","of","hidden-states","at","the","output","of","the","last","layer","of","the","model",".","*","*","pooler_output","*","*",":","``","torch.FloatTensor","``","of","shape","``","(","batch_size",",","hidden_size",")","``",".","Last","layer","hidden-state","of","the","first","token","of","the","sequence","(","classification","token",")","further","processed","by","a","Linear","layer","and","a","Tanh","activation","function",".","The","Linear","layer","weights","are","trained","from","the","next","sentence","prediction","(","classification",")","objective","during","Bert","pretraining",".","This","output","is","usually","*","not","*","a","good","summary","of","the","semantic","content","of","the","input",",","you","'re","often","better","with","averaging","or","pooling","the","sequence","of","hidden-states","for","the","whole","input","sequence",".","*","*","hidden_states","*","*",":","(","`","optional","`",",","returned","when","``","config.output_hidden_states=True","``",")","list","of","``","torch.FloatTensor","``","(","one","for","the","output","of","each","layer","+","the","output","of","the","embeddings",")","of","shape","``","(","batch_size",",","sequence_length",",","hidden_size",")","``",":","Hidden-states","of","the","model","at","the","output","of","each","layer","plus","the","initial","embedding","outputs",".","*","*","attentions","*","*",":","(","`","optional","`",",","returned","when","``","config.output_attentions=True","``",")","list","of","``","torch.FloatTensor","``","(","one","for","each","layer",")","of","shape","``","(","batch_size",",","num_heads",",","sequence_length",",","sequence_length",")","``",":","Attentions","weights","after","the","attention","softmax",",","used","to","compute","the","weighted","average","in","the","self-attention","heads",".","Examples",":",":","#","For","example","purposes",".","Not","runnable",".","transformer","=","BertModel.from_pretrained","(","'bert-base-uncased","'",")","encoder","=","ImageEncoder","(","args",")","mmbt","=","MMBTModel","(","config",",","transformer",",","encoder",")"]
code: def forward(
      self,
      input_modal: Tensor,
      input_ids: Tensor,
      modal_start_tokens: Optional[Tensor] = None,
      modal_end_tokens: Optional[Tensor] = None,
      attention_mask: Optional[Tensor] = None,
      token_type_ids: Optional[Tensor] = None,
      modal_token_type_ids: Optional[Tensor] = None,
      position_ids: Optional[Tensor] = None,
      modal_position_ids: Optional[Tensor] = None,
      head_mask: Optional[Tensor] = None,
      inputs_embeds: Optional[Tensor] = None,
      encoder_hidden_states: Optional[Tensor] = None,
      encoder_attention_mask: Optional[Tensor] = None,
  ):
      if input_ids is not None and inputs_embeds is not None:
          raise ValueError(
              "You cannot specify both input_ids and inputs_embeds at the same time"
          )
      elif input_ids is not None:
          input_txt_shape = input_ids.size()
      elif inputs_embeds is not None:
          input_txt_shape = inputs_embeds.size()[:-1]
      else:
          raise ValueError(
              "You have to specify either input_ids or inputs_embeds"
          )
  
      device = (
          inputs_embeds.device
          if inputs_embeds is not None
          else input_ids.device
      )
  
      modal_embeddings = self.modal_encoder(
          input_modal,
          start_token=modal_start_tokens,
          end_token=modal_end_tokens,
          position_ids=modal_position_ids,
          token_type_ids=modal_token_type_ids,
      )
  
      input_modal_shape = modal_embeddings.size()[:-1]
  
      if token_type_ids is None:
          token_type_ids = torch.ones(
              input_txt_shape, dtype=torch.long, device=device
          )
  
      txt_embeddings = self.transformer.embeddings(
          input_ids=input_ids,
          position_ids=position_ids,
          token_type_ids=token_type_ids,
          inputs_embeds=inputs_embeds,
      )
  
      embedding_output = torch.cat(
          [modal_embeddings, txt_embeddings], 1
      )
  
      input_shape = embedding_output.size()[:-1]
  
      if attention_mask is None:
          attention_mask = torch.ones(input_shape, device=device)
      else:
          attention_mask = torch.cat(
              [
                  torch.ones(
                      input_modal_shape, device=device, dtype=torch.long
                  ),
                  attention_mask,
              ],
              dim=1,
          )
  
      if encoder_attention_mask is None:
          encoder_attention_mask = torch.ones(
              input_shape, device=device
          )
      else:
          encoder_attention_mask = torch.cat(
              [
                  torch.ones(input_modal_shape, device=device),
                  encoder_attention_mask,
              ],
              dim=1,
          )
  
      # We can provide a self-attention mask of dimensions
      # [batch_size, from_seq_length, to_seq_length]
      # ourselves in which case we just need to make it broadcastable to all heads.
      if attention_mask.dim() == 3:
          attention_mask = attention_mask[:, None, :, :]
  
      # Provided a padding mask of dimensions [batch_size, seq_length]
      # - if the model is a decoder, apply a causal mask in addition to the
      #   padding mask
      # - if the model is an encoder, make the mask broadcastable to
      # [batch_size, num_heads, seq_length, seq_length]
      if attention_mask.dim() == 2:
          if self.is_decoder:
              batch_size, seq_length = input_shape
              seq_ids = torch.arange(seq_length, device=device)
              causal_mask = (
                  seq_ids[None, None, :].repeat(
                      batch_size, seq_length, 1
                  )
                  <= seq_ids[None, :, None]
              )
              attention_mask = (
                  causal_mask[:, None, :, :]
                  * attention_mask[:, None, None, :]
              )
          else:
              attention_mask = attention_mask[:, None, None, :]
  
      # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
      # masked positions, this operation will create a tensor which is 0.0 for
      # positions we want to attend and -10000.0 for masked positions.
      # Since we are adding it to the raw scores before the softmax, this is
      # effectively the same as removing these entirely.
      # Python builtin next is currently not supported in Torchscript
      if not torch.jit.is_scripting():
          attention_mask = attention_mask.to(
              dtype=next(self.parameters()).dtype
          )  # fp16 compatibility
      attention_mask = (1.0 - attention_mask) * -10000.0
  
      # If a 2D ou 3D attention mask is provided for the cross-attention
      # we need to make broadcastabe to
      # [batch_size, num_heads, seq_length, seq_length]
      if encoder_attention_mask.dim() == 3:
          encoder_attention_mask = encoder_attention_mask[:, None, :, :]
      if encoder_attention_mask.dim() == 2:
          encoder_attention_mask = encoder_attention_mask[
              :, None, None, :
          ]
  
      # Python builtin next is currently not supported in Torchscript
      if not torch.jit.is_scripting():
          encoder_attention_mask = encoder_attention_mask.to(
              dtype=next(self.parameters()).dtype
          )  # fp16 compatibility
  
      encoder_attention_mask = (1.0 - encoder_attention_mask) * -10000.0
  
      encoder_outputs = self.transformer.encoder(
          embedding_output,
          attention_mask=attention_mask,
          encoder_hidden_states=encoder_hidden_states,
          encoder_attention_mask=encoder_attention_mask,
      )
  
      sequence_output = encoder_outputs[0]
      pooled_output = self.transformer.pooler(sequence_output)
  
      outputs = (
          sequence_output,
          pooled_output,
          encoder_outputs[1:],
      )  # add hidden_states and attentions if they are here
      return outputs
  
imports: import os
  from copy import deepcopy
  from dataclasses import dataclass
  from typing import Dict, Optional, Union
  import torch
  from mmf.common.registry import registry
  from mmf.models.base_model import BaseModel
  from mmf.models.interfaces.mmbt import MMBTGridHMInterface
  from mmf.modules.encoders import EncoderFactory, ImageEncoderFactory, ImageEncoderTypes, MultiModalEncoderBase, ResNet152ImageEncoder, TextEncoderFactory, TextEncoderTypes, TransformerEncoder
  from mmf.modules.hf_layers import replace_with_jit
  from mmf.utils.checkpoint import load_pretrained_model
  from mmf.utils.configuration import get_mmf_cache_dir
  from mmf.utils.modeling import get_optimizer_parameters_for_bert
  from omegaconf import DictConfig, II, OmegaConf
  from torch import nn, Tensor
label_desc: Use image node_id 2 for calling the MMBTModel obj's underlying member method code with example usage: obj.forward(input_modal, input_ids, modal_start_tokens, modal_end_tokens, attention_mask, token_type_ids, modal_token_type_ids, position_ids, modal_position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask) and returns: outputs