class_docstr: Uses a pretrained model from `transformers` as a `TokenEmbedder`.
  
  Registered as a `TokenEmbedder` with name "pretrained_transformer".
  
  # Parameters
  
  model_name : `str`
      The name of the `transformers` model to use. Should be the same as the corresponding
      `PretrainedTransformerIndexer`.
  max_length : `int`, optional (default = `None`)
      If positive, folds input token IDs into multiple segments of this length, pass them
      through the transformer model independently, and concatenate the final representations.
      Should be set to the same value as the `max_length` option on the
      `PretrainedTransformerIndexer`.
  sub_module: `str`, optional (default = `None`)
      The name of a submodule of the transformer to be used as the embedder. Some transformers naturally act
      as embedders such as BERT. However, other models consist of encoder and decoder, in which case we just
      want to use the encoder.
  train_parameters: `bool`, optional (default = `True`)
      If this is `True`, the transformer weights get updated during training. If this is `False`, the
      transformer weights are not updated during training.
  eval_mode: `bool`, optional (default = `False`)
      If this is `True`, the model is always set to evaluation mode (e.g., the dropout is disabled and the
      batch normalization layer statistics are not updated). If this is `False`, such dropout and batch
      normalization layers are only set to evaluation mode when when the model is evaluating on development
      or test data.
  last_layer_only: `bool`, optional (default = `True`)
      When `True` (the default), only the final layer of the pretrained transformer is taken
      for the embeddings. But if set to `False`, a scalar mix of all of the layers
      is used.
  override_weights_file: `Optional[str]`, optional (default = `None`)
      If set, this specifies a file from which to load alternate weights that override the
      weights from huggingface. The file is expected to contain a PyTorch `state_dict`, created
      with `torch.save()`.
  override_weights_strip_prefix: `Optional[str]`, optional (default = `None`)
      If set, strip the given prefix from the state dict when loading it.
  reinit_modules: `Optional[Union[int, Tuple[int, ...], Tuple[str, ...]]]`, optional (default = `None`)
      If this is an integer, the last `reinit_modules` layers of the transformer will be
      re-initialized. If this is a tuple of integers, the layers indexed by `reinit_modules` will
      be re-initialized. Note, because the module structure of the transformer `model_name` can
      differ, we cannot guarantee that providing an integer or tuple of integers will work. If
      this fails, you can instead provide a tuple of strings, which will be treated as regexes and
      any module with a name matching the regex will be re-initialized. Re-initializing the last
      few layers of a pretrained transformer can reduce the instability of fine-tuning on small
      datasets and may improve performance (https://arxiv.org/abs/2006.05987v3). Has no effect
      if `load_weights` is `False` or `override_weights_file` is not `None`.
  load_weights: `bool`, optional (default = `True`)
      Whether to load the pretrained weights. If you're loading your model/predictor from an AllenNLP archive
      it usually makes sense to set this to `False` (via the `overrides` parameter)
      to avoid unnecessarily caching and loading the original pretrained weights,
      since the archive will already contain all of the weights needed.
  gradient_checkpointing: `bool`, optional (default = `None`)
      Enable or disable gradient checkpointing.
  tokenizer_kwargs: `Dict[str, Any]`, optional (default = `None`)
      Dictionary with
      [additional arguments](https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_utils.py#L691)
      for `AutoTokenizer.from_pretrained`.
  transformer_kwargs: `Dict[str, Any]`, optional (default = `None`)
      Dictionary with
      [additional arguments](https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/modeling_utils.py#L253)
      for `AutoModel.from_pretrained`.
class_docstr_tok: ["Uses","a","pretrained","model","from","`","transformers","`","as","a","`","TokenEmbedder","`",".","Registered","as","a","`","TokenEmbedder","`","with","name","``","pretrained_transformer","''",".","#","Parameters","model_name",":","`","str","`","The","name","of","the","`","transformers","`","model","to","use",".","Should","be","the","same","as","the","corresponding","`","PretrainedTransformerIndexer","`",".","max_length",":","`","int","`",",","optional","(","default","=","`","None","`",")","If","positive",",","folds","input","token","IDs","into","multiple","segments","of","this","length",",","pass","them","through","the","transformer","model","independently",",","and","concatenate","the","final","representations",".","Should","be","set","to","the","same","value","as","the","`","max_length","`","option","on","the","`","PretrainedTransformerIndexer","`",".","sub_module",":","`","str","`",",","optional","(","default","=","`","None","`",")","The","name","of","a","submodule","of","the","transformer","to","be","used","as","the","embedder",".","Some","transformers","naturally","act","as","embedders","such","as","BERT",".","However",",","other","models","consist","of","encoder","and","decoder",",","in","which","case","we","just","want","to","use","the","encoder",".","train_parameters",":","`","bool","`",",","optional","(","default","=","`","True","`",")","If","this","is","`","True","`",",","the","transformer","weights","get","updated","during","training",".","If","this","is","`","False","`",",","the","transformer","weights","are","not","updated","during","training",".","eval_mode",":","`","bool","`",",","optional","(","default","=","`","False","`",")","If","this","is","`","True","`",",","the","model","is","always","set","to","evaluation","mode","(","e.g.",",","the","dropout","is","disabled","and","the","batch","normalization","layer","statistics","are","not","updated",")",".","If","this","is","`","False","`",",","such","dropout","and","batch","normalization","layers","are","only","set","to","evaluation","mode","when","when","the","model","is","evaluating","on","development","or","test","data",".","last_layer_only",":","`","bool","`",",","optional","(","default","=","`","True","`",")","When","`","True","`","(","the","default",")",",","only","the","final","layer","of","the","pretrained","transformer","is","taken","for","the","embeddings",".","But","if","set","to","`","False","`",",","a","scalar","mix","of","all","of","the","layers","is","used",".","override_weights_file",":","`","Optional","[","str","]","`",",","optional","(","default","=","`","None","`",")","If","set",",","this","specifies","a","file","from","which","to","load","alternate","weights","that","override","the","weights","from","huggingface",".","The","file","is","expected","to","contain","a","PyTorch","`","state_dict","`",",","created","with","`","torch.save","(",")","`",".","override_weights_strip_prefix",":","`","Optional","[","str","]","`",",","optional","(","default","=","`","None","`",")","If","set",",","strip","the","given","prefix","from","the","state","dict","when","loading","it",".","reinit_modules",":","`","Optional","[","Union","[","int",",","Tuple","[","int",",","...","]",",","Tuple","[","str",",","...","]","]","]","`",",","optional","(","default","=","`","None","`",")","If","this","is","an","integer",",","the","last","`","reinit_modules","`","layers","of","the","transformer","will","be","re-initialized",".","If","this","is","a","tuple","of","integers",",","the","layers","indexed","by","`","reinit_modules","`","will","be","re-initialized",".","Note",",","because","the","module","structure","of","the","transformer","`","model_name","`","can","differ",",","we","can","not","guarantee","that","providing","an","integer","or","tuple","of","integers","will","work",".","If","this","fails",",","you","can","instead","provide","a","tuple","of","strings",",","which","will","be","treated","as","regexes","and","any","module","with","a","name","matching","the","regex","will","be","re-initialized",".","Re-initializing","the","last","few","layers","of","a","pretrained","transformer","can","reduce","the","instability","of","fine-tuning","on","small","datasets","and","may","improve","performance","(","https",":","\/\/arxiv.org\/abs\/2006.05987v3",")",".","Has","no","effect","if","`","load_weights","`","is","`","False","`","or","`","override_weights_file","`","is","not","`","None","`",".","load_weights",":","`","bool","`",",","optional","(","default","=","`","True","`",")","Whether","to","load","the","pretrained","weights",".","If","you","'re","loading","your","model\/predictor","from","an","AllenNLP","archive","it","usually","makes","sense","to","set","this","to","`","False","`","(","via","the","`","overrides","`","parameter",")","to","avoid","unnecessarily","caching","and","loading","the","original","pretrained","weights",",","since","the","archive","will","already","contain","all","of","the","weights","needed",".","gradient_checkpointing",":","`","bool","`",",","optional","(","default","=","`","None","`",")","Enable","or","disable","gradient","checkpointing",".","tokenizer_kwargs",":","`","Dict","[","str",",","Any","]","`",",","optional","(","default","=","`","None","`",")","Dictionary","with","[","additional","arguments","]","(","https",":","\/\/github.com\/huggingface\/transformers\/blob\/155c782a2ccd103cf63ad48a2becd7c76a7d2115\/transformers\/tokenization_utils.py","#","L691",")","for","`","AutoTokenizer.from_pretrained","`",".","transformer_kwargs",":","`","Dict","[","str",",","Any","]","`",",","optional","(","default","=","`","None","`",")","Dictionary","with","[","additional","arguments","]","(","https",":","\/\/github.com\/huggingface\/transformers\/blob\/155c782a2ccd103cf63ad48a2becd7c76a7d2115\/transformers\/modeling_utils.py","#","L253",")","for","`","AutoModel.from_pretrained","`","."]
docstr: We take 2D segments of a long sequence and flatten them out to get the whole sequence
  representation while remove unnecessary special tokens.
  
  [ [ [CLS]_emb A_emb B_emb C_emb [SEP]_emb ], [ [CLS]_emb D_emb E_emb [SEP]_emb [PAD]_emb ] ]
  -> [ [CLS]_emb A_emb B_emb C_emb D_emb E_emb [SEP]_emb ]
  
  We truncate the start and end tokens for all segments, recombine the segments,
  and manually add back the start and end tokens.
  
  # Parameters
  
  embeddings: `torch.FloatTensor`
      Shape: [batch_size * num_segments, self._max_length, embedding_size].
  mask: `torch.BoolTensor`
      Shape: [batch_size * num_segments, self._max_length].
      The mask for the concatenated segments of wordpieces. The same as `segment_concat_mask`
      in `forward()`.
  batch_size: `int`
  num_segment_concat_wordpieces: `int`
      The length of the original "[ [CLS] A B C [SEP] [CLS] D E F [SEP] ]", i.e.
      the original `token_ids.size(1)`.
  
  # Returns:
  
  embeddings: `torch.FloatTensor`
      Shape: [batch_size, self._num_wordpieces, embedding_size].
docstr_tok: ["We","take","2D","segments","of","a","long","sequence","and","flatten","them","out","to","get","the","whole","sequence","representation","while","remove","unnecessary","special","tokens",".","[","[","[","CLS","]","_emb","A_emb","B_emb","C_emb","[","SEP","]","_emb","]",",","[","[","CLS","]","_emb","D_emb","E_emb","[","SEP","]","_emb","[","PAD","]","_emb","]","]","-",">","[","[","CLS","]","_emb","A_emb","B_emb","C_emb","D_emb","E_emb","[","SEP","]","_emb","]","We","truncate","the","start","and","end","tokens","for","all","segments",",","recombine","the","segments",",","and","manually","add","back","the","start","and","end","tokens",".","#","Parameters","embeddings",":","`","torch.FloatTensor","`","Shape",":","[","batch_size","*","num_segments",",","self._max_length",",","embedding_size","]",".","mask",":","`","torch.BoolTensor","`","Shape",":","[","batch_size","*","num_segments",",","self._max_length","]",".","The","mask","for","the","concatenated","segments","of","wordpieces",".","The","same","as","`","segment_concat_mask","`","in","`","forward","(",")","`",".","batch_size",":","`","int","`","num_segment_concat_wordpieces",":","`","int","`","The","length","of","the","original","``","[","[","CLS","]","A","B","C","[","SEP","]","[","CLS","]","D","E","F","[","SEP","]","]","''",",","i.e",".","the","original","`","token_ids.size","(","1",")","`",".","#","Returns",":","embeddings",":","`","torch.FloatTensor","`","Shape",":","[","batch_size",",","self._num_wordpieces",",","embedding_size","]","."]
code: def _unfold_long_sequences(
      self,
      embeddings: torch.FloatTensor,
      mask: torch.BoolTensor,
      batch_size: int,
      num_segment_concat_wordpieces: int,
  ) -> torch.FloatTensor:
      """
      We take 2D segments of a long sequence and flatten them out to get the whole sequence
      representation while remove unnecessary special tokens.
  
      [ [ [CLS]_emb A_emb B_emb C_emb [SEP]_emb ], [ [CLS]_emb D_emb E_emb [SEP]_emb [PAD]_emb ] ]
      -> [ [CLS]_emb A_emb B_emb C_emb D_emb E_emb [SEP]_emb ]
  
      We truncate the start and end tokens for all segments, recombine the segments,
      and manually add back the start and end tokens.
  
      # Parameters
  
      embeddings: `torch.FloatTensor`
          Shape: [batch_size * num_segments, self._max_length, embedding_size].
      mask: `torch.BoolTensor`
          Shape: [batch_size * num_segments, self._max_length].
          The mask for the concatenated segments of wordpieces. The same as `segment_concat_mask`
          in `forward()`.
      batch_size: `int`
      num_segment_concat_wordpieces: `int`
          The length of the original "[ [CLS] A B C [SEP] [CLS] D E F [SEP] ]", i.e.
          the original `token_ids.size(1)`.
  
      # Returns:
  
      embeddings: `torch.FloatTensor`
          Shape: [batch_size, self._num_wordpieces, embedding_size].
      """
  
      def lengths_to_mask(lengths, max_len, device):
          return torch.arange(max_len, device=device).expand(
              lengths.size(0), max_len
          ) < lengths.unsqueeze(1)
  
      device = embeddings.device
      num_segments = int(embeddings.size(0) / batch_size)
      embedding_size = embeddings.size(2)
  
      # We want to remove all segment-level special tokens but maintain sequence-level ones
      num_wordpieces = (
          num_segment_concat_wordpieces
          - (num_segments - 1) * self._num_added_tokens
      )
  
      embeddings = embeddings.reshape(
          batch_size, num_segments * self._max_length, embedding_size  # type: ignore
      )
      mask = mask.reshape(batch_size, num_segments * self._max_length)  # type: ignore
      # We assume that all 1s in the mask precede all 0s, and add an assert for that.
      # Open an issue on GitHub if this breaks for you.
      # Shape: (batch_size,)
      seq_lengths = mask.sum(-1)
      if not (
          lengths_to_mask(seq_lengths, mask.size(1), device) == mask
      ).all():
          raise ValueError(
              "Long sequence splitting only supports masks with all 1s preceding all 0s."
          )
      # Shape: (batch_size, self._num_added_end_tokens); this is a broadcast op
      end_token_indices = (
          seq_lengths.unsqueeze(-1)
          - torch.arange(self._num_added_end_tokens, device=device)
          - 1
      )
  
      # Shape: (batch_size, self._num_added_start_tokens, embedding_size)
      start_token_embeddings = embeddings[
          :, : self._num_added_start_tokens, :
      ]
      # Shape: (batch_size, self._num_added_end_tokens, embedding_size)
      end_token_embeddings = batched_index_select(
          embeddings, end_token_indices
      )
  
      embeddings = embeddings.reshape(
          batch_size, num_segments, self._max_length, embedding_size
      )
      embeddings = embeddings[
          :,
          :,
          self._num_added_start_tokens : embeddings.size(2)
          - self._num_added_end_tokens,
          :,
      ]  # truncate segment-level start/end tokens
      embeddings = embeddings.reshape(
          batch_size, -1, embedding_size
      )  # flatten
  
      # Now try to put end token embeddings back which is a little tricky.
  
      # The number of segment each sequence spans, excluding padding. Mimicking ceiling operation.
      # Shape: (batch_size,)
      num_effective_segments = (
          seq_lengths + self._max_length - 1
      ) // self._max_length
      # The number of indices that end tokens should shift back.
      num_removed_non_end_tokens = (
          num_effective_segments * self._num_added_tokens
          - self._num_added_end_tokens
      )
      # Shape: (batch_size, self._num_added_end_tokens)
      end_token_indices -= num_removed_non_end_tokens.unsqueeze(-1)
      assert (end_token_indices >= self._num_added_start_tokens).all()
      # Add space for end embeddings
      embeddings = torch.cat(
          [embeddings, torch.zeros_like(end_token_embeddings)], 1
      )
      # Add end token embeddings back
      embeddings.scatter_(
          1,
          end_token_indices.unsqueeze(-1).expand_as(
              end_token_embeddings
          ),
          end_token_embeddings,
      )
  
      # Now put back start tokens. We can do this before putting back end tokens, but then
      # we need to change `num_removed_non_end_tokens` a little.
      embeddings = torch.cat([start_token_embeddings, embeddings], 1)
  
      # Truncate to original length
      embeddings = embeddings[:, :num_wordpieces, :]
      return embeddings
  
file_path: allennlp/allennlp/modules/token_embedders/pretrained_transformer_embedder.py
imports: import logging
  import math
  from typing import Any, Dict, Optional, Tuple, Union
  import torch
  import torch.nn.functional
  from allennlp.data.tokenizers import PretrainedTransformerTokenizer
  from allennlp.modules.scalar_mix import ScalarMix
  from allennlp.modules.token_embedders.token_embedder import TokenEmbedder
  from allennlp.nn.util import batched_index_select
  from transformers import XLNetConfig
label_desc: Use image node_id 7 for calling the PretrainedTransformerEmbedder obj's underlying member method code with example usage: obj._unfold_long_sequences(embeddings, mask, batch_size, num_segment_concat_wordpieces) and returns: embeddings, unknown