class_docstr: Implementation of the Transformer Sentence Encoder. This directly makes
  use of the TransformerSentenceEncoder module in Fairseq.
  
  A few interesting config options:
      - encoder_normalize_before detemines whether the layer norm is applied
        before or after self_attention. This is similar to original
        implementation from Google.
      - activation_fn can be set to 'gelu' instead of the default of 'relu'.
code: def upgrade_state_dict_named(self, state_dict):
      # We convert in_proj_weight to individual q,k,v weights
      items_to_add = {}
      keys_to_remove = []
      for k in state_dict.keys():
          if k.endswith("in_proj_weight"):
              # in_proj_weight used to be q + k + v with same dimensions
              dim = int(state_dict[k].shape[0] / 3)
              items_to_add[
                  k.replace("in_proj_weight", "q_proj.weight")
              ] = state_dict[k][:dim]
              items_to_add[
                  k.replace("in_proj_weight", "k_proj.weight")
              ] = state_dict[k][dim : 2 * dim]
              items_to_add[
                  k.replace("in_proj_weight", "v_proj.weight")
              ] = state_dict[k][2 * dim :]
              keys_to_remove.append(k)
  
          if k.endswith("in_proj_bias"):
              dim = int(state_dict[k].shape[0] / 3)
              items_to_add[
                  k.replace("in_proj_bias", "q_proj.bias")
              ] = state_dict[k][:dim]
              items_to_add[
                  k.replace("in_proj_bias", "k_proj.bias")
              ] = state_dict[k][dim : 2 * dim]
              items_to_add[
                  k.replace("in_proj_bias", "v_proj.bias")
              ] = state_dict[k][2 * dim :]
              keys_to_remove.append(k)
  
      for k in keys_to_remove:
          del state_dict[k]
  
      for key, value in items_to_add.items():
          state_dict[key] = value
  
      return state_dict
  
file_path: pytext/pytext/models/representations/transformer_sentence_encoder.py
imports: from typing import Tuple
  import torch
  from fairseq.modules import TransformerSentenceEncoder
  from pytext.config import ConfigBase
  from pytext.models.representations.traced_transformer_encoder import TracedTransformerEncoder
  from pytext.models.representations.transformer_sentence_encoder_base import TransformerSentenceEncoderBase
  from pytext.utils.usage import log_class_usage
label_desc: Use image node_id 5 for calling the TransformerSentenceEncoder obj's underlying member method code with example usage: obj.upgrade_state_dict_named(state_dict) and returns: state_dict