class_docstr: Definitely can represent BERT, but also a lot of other things. To be used for MLM schemes.
class_docstr_tok: ["Definitely","can","represent","BERT",",","but","also","a","lot","of","other","things",".","To","be","used","for","MLM","schemes","."]
code: def __init__(self, cfg_arch):
      super().__init__()
      self.cfg = cfg_arch
  
      self.embedding = EmbeddingComponent(
          cfg_arch.embedding, cfg_arch.norm, cfg_arch.norm_eps
      )
      if cfg_arch.embedding.embedding_dim == cfg_arch.hidden_size:
          self.input_projection = torch.nn.Identity()
      else:
          self.input_projection = torch.nn.Linear(
              cfg_arch.embedding.embedding_dim,
              cfg_arch.hidden_size,
              bias=cfg_arch.use_bias,
          )
  
      self.recurrent_layer = SequentialwithMask(
          [
              TransformerLayer(idx, cfg_arch)
              for idx in range(cfg_arch.recurrent_layers)
          ]
      )
  
      self.seq_first = (
          self.recurrent_layer.LAYOUT == "[S B H]"
          if len(self.recurrent_layer) > 0
          else False
      )
  
imports: import torch
  from typing import Optional
  from random import randrange
  from transformers import PretrainedConfig
  from .components import EmbeddingComponent, AttentionComponent, FFNComponent, PredictionHeadComponent, get_extended_attention_mask, _get_norm_fn, _get_nonlin_fn, _init_module
label_desc: Use image node_id 1 to create a new ScriptableRecurrentLM object from inherited base classes: torch.nn with example: obj = ScriptableRecurrentLM(cfg_arch)