class_docstr: Implementation of an AdapterFusion block.
code: def forward(
      self, query, key, value, residual, output_attentions: bool = False
  ):
      if self.config["residual_before"]:
          value += residual[:, :, None, :].repeat(
              1, 1, value.size(2), 1
          )
  
      if self.config["query"]:
          query_layer = self.query(query)
      else:
          query_layer = query
  
      if self.config["key"]:
          key_layer = self.key(key)
      else:
          key_layer = key
  
      if self.config["value"] and self.config["value_before_softmax"]:
          # key/value have dims => batch, toks, number-of-adapters, feats
          value_layer = self.value(value)
      else:
          value_layer = value
  
      # Take the dot product between "query" and "key" to get the raw attention scores.
      attention_scores = torch.squeeze(
          torch.matmul(
              query_layer.unsqueeze(2), key_layer.transpose(-2, -1)
          ),
          dim=2,
      )
  
      attention_scores = self.dropout(attention_scores)
  
      # Normalize the attention scores to probabilities.
      attention_probs = nn.Softmax(dim=-1)(attention_scores / self.T)
      self.T = max(self.T - self.reduction, 1.0)
  
      context_layer = torch.squeeze(
          torch.matmul(attention_probs.unsqueeze(2), value_layer), dim=2
      )
  
      if (
          self.config["value"]
          and not self.config["value_before_softmax"]
      ):
          # key/value have dims => batch, toks, number-of-adapters, feats
          context_layer = self.value(context_layer)
      else:
          context_layer = context_layer
  
      if not self.config["residual_before"]:
          context_layer += residual
  
      if output_attentions:
          attention_probs = attention_probs.detach().cpu().numpy()
          return context_layer, attention_probs
      else:
          return context_layer
  
imports: import math
  import torch
  from torch import nn
  from transformers.activations import get_activation
  from ..configuration import AdapterFusionConfig, BnConfig
  from ..context import ForwardContext
label_desc: Use image node_id 2 for calling the BertFusion obj's underlying member method code with example usage: obj.forward(query, key, value, residual, output_attentions) and returns: context_layer,  attention_probs, context_layer