class_docstr: A module that directly utilizes the pretrained models in `transformers`_ to produce BERT representations.
  
  While mainly tailored to provide input preparation and post-processing for the BERT model,
  it is also compatiable with other pretrained language models like XLNet, RoBERTa and ELECTRA, etc.
  
  Args:
      model (str):
          Path or name of the pretrained models registered in `transformers`_, e.g., ``'bert-base-cased'``.
      n_layers (int):
          The number of layers from the model to use.
          If 0, uses all layers.
      n_out (int):
          The requested size of the embeddings.
          If 0, uses the size of the pretrained embedding model.
      pad_index (int):
          The index of the padding token in the BERT vocabulary. Default: 0.
      max_len (int):
          Sequences should not exceed the specfied max length. Default: 512.
      dropout (float):
          The dropout ratio of BERT layers. Default: 0.
          This value will be passed into the :class:`ScalarMix` layer.
      requires_grad (bool):
          If ``True``, the model parameters will be updated together with the downstream task.
          Default: ``False``.
  
  .. _transformers:
      https://github.com/huggingface/transformers
docstr: Args:
      subwords (~torch.Tensor): ``[batch_size, seq_len, fix_len]``.
  
  Returns:
      ~torch.Tensor:
          BERT embeddings of shape ``[batch_size, seq_len, n_out]``.
docstr_tok: ["Args",":","subwords","(","~torch.Tensor",")",":","``","[","batch_size",",","seq_len",",","fix_len","]","``",".","Returns",":","~torch.Tensor",":","BERT","embeddings","of","shape","``","[","batch_size",",","seq_len",",","n_out","]","``","."]
code: def forward(self, subwords):
      r"""
      Args:
          subwords (~torch.Tensor): ``[batch_size, seq_len, fix_len]``.
  
      Returns:
          ~torch.Tensor:
              BERT embeddings of shape ``[batch_size, seq_len, n_out]``.
      """
      batch_size, seq_len, fix_len = subwords.shape
      if self.max_len and seq_len > self.max_len:
          raise RuntimeError(
              f"Token indices sequence length is longer than the specified max length "
              f"({seq_len} > {self.max_len})"
          )
  
      mask = subwords.ne(self.pad_index)
      lens = mask.sum((1, 2))
      # [batch_size, n_subwords]
      subwords = pad_sequence(subwords[mask].split(lens.tolist()), True)
      bert_mask = pad_sequence(mask[mask].split(lens.tolist()), True)
      # return the hidden states of all layers
      bert = self.bert(subwords, attention_mask=bert_mask.float())[-1]
      # [n_layers, batch_size, n_subwords, hidden_size]
      bert = bert[-self.n_layers :]
      # [batch_size, n_subwords, hidden_size]
      bert = self.scalar_mix(bert)
      # [batch_size, n_subwords]
      bert_lens = mask.sum(-1)
      bert_lens = bert_lens.masked_fill_(bert_lens.eq(0), 1)
      # [batch_size, seq_len, fix_len, hidden_size]
      embed = bert.new_zeros(*mask.shape, self.hidden_size)
      embed = embed.masked_scatter_(mask.unsqueeze(-1), bert[bert_mask])
      # [batch_size, seq_len, hidden_size]
      embed = embed.sum(2) / bert_lens.unsqueeze(-1)
      embed = self.projection(embed)
  
      return embed
  
file_path: underthesea/underthesea/modules/bert.py
imports: import torch.nn
  from torch.nn.utils.rnn import pad_sequence
  from transformers import AutoConfig, AutoModel
  from underthesea.modules.scalar_mix import ScalarMix
label_desc: Use image node_id 3 for calling the BertEmbedding obj's underlying member method code with example usage: obj.forward(subwords) and returns: embed