class_docstr: Configuration for `BertModel`.
class_docstr_tok: ["Configuration","for","`","BertModel","`","."]
docstr: Constructs BertConfig.
  
  residual_structure="original"
     TODO(noam): describe
  residual_structure="direct"
     TODO(noam): describe
  
  
  Args:
    vocab_size: Vocabulary size of `inputs_ids` in `BertModel`.
    d_model: Number of channels in input/output of each layer.
    position_signal: A string specifying the type of position signal.
      Implemented values are "embedding", "relative_attention_bias".
    max_position_embeddings: For models using positional embeddings,
      this is the maximum sequence length that this model might
      ever be used with. Typically set this to something large just in case
      (e.g., 512 or 1024 or 2048).
    num_blocks: Number of (attention+feed-forward) blocks in the Transformer
       encoder.
    block_layers: a comma-separated string specifying the sequence of layers
      in each block.
    layer_output_dropout_prob: The dropout probability for the output of
      each layer.
    residual_structure: a string.  Legal values are "original" and "direct".
    use_bias: a boolean - If true, then we use biases for dense layers and
      in layer normalization, and subtract off the mean in layer
      normalization.
    attention_num_heads: Number of attention heads for each attention layer in
      the Transformer encoder.
    attention_head_size: Size of attention keys and values.  If set to None,
      a default value is used equal to (d_model / attention_num_heads)
    attention_num_key_heads: Number of attention key heads.
    attention_key_head_size: Size of attention keys.
    attention_num_value_heads: Number of attention value heads.
    attention_value_head_size: Size of attention values.
    attention_probs_dropout_prob: The dropout ratio for the attention
      probabilities.
    feedforward_intermediate_size: The size of the "intermediate" layer in the
       feed-forward layer in the Transformer encoder (a.k.a. d_ff).
    feedforward_intermediate_act: The non-linear activation function
      (function or string) applied to the feedforward intermediate layer
      and the pooler layer.
    feedforward_intermediate_dropout_prob: The dropout probability for
      feed-forward intermediate layer.
    moe_num_experts: an integer - number of experts in moe layer
    moe_intermediate_size: an integer - size of intermediate layer in each
      expert
    type_vocab_size: The vocabulary size of the `token_type_ids` passed into
      `BertModel`.
    initializer_range: The stdev of the truncated_normal_initializer for
      initializing all weight matrices.
code: def __init__(
      self,
      vocab_size,
      d_model=768,
      position_signal="embedding",
      max_position_embeddings=512,
      num_blocks=12,
      block_layers="attention,feedforward",
      layer_output_dropout_prob=0.1,
      residual_structure="original",
      use_bias=True,
      attention_num_heads=12,
      attention_head_size=None,
      attention_num_key_heads=None,
      attention_key_head_size=None,
      attention_num_value_heads=None,
      attention_value_head_size=None,
      attention_probs_dropout_prob=0.1,
      feedforward_intermediate_size=3072,
      feedforward_intermediate_act="gelu",
      feedforward_intermediate_dropout_prob=0.0,
      moe_num_experts=32,
      moe_intermediate_size=6144,
      type_vocab_size=16,
      initializer_range=0.02,
  ):
      """Constructs BertConfig.
  
      residual_structure="original"
         TODO(noam): describe
      residual_structure="direct"
         TODO(noam): describe
  
  
      Args:
        vocab_size: Vocabulary size of `inputs_ids` in `BertModel`.
        d_model: Number of channels in input/output of each layer.
        position_signal: A string specifying the type of position signal.
          Implemented values are "embedding", "relative_attention_bias".
        max_position_embeddings: For models using positional embeddings,
          this is the maximum sequence length that this model might
          ever be used with. Typically set this to something large just in case
          (e.g., 512 or 1024 or 2048).
        num_blocks: Number of (attention+feed-forward) blocks in the Transformer
           encoder.
        block_layers: a comma-separated string specifying the sequence of layers
          in each block.
        layer_output_dropout_prob: The dropout probability for the output of
          each layer.
        residual_structure: a string.  Legal values are "original" and "direct".
        use_bias: a boolean - If true, then we use biases for dense layers and
          in layer normalization, and subtract off the mean in layer
          normalization.
        attention_num_heads: Number of attention heads for each attention layer in
          the Transformer encoder.
        attention_head_size: Size of attention keys and values.  If set to None,
          a default value is used equal to (d_model / attention_num_heads)
        attention_num_key_heads: Number of attention key heads.
        attention_key_head_size: Size of attention keys.
        attention_num_value_heads: Number of attention value heads.
        attention_value_head_size: Size of attention values.
        attention_probs_dropout_prob: The dropout ratio for the attention
          probabilities.
        feedforward_intermediate_size: The size of the "intermediate" layer in the
           feed-forward layer in the Transformer encoder (a.k.a. d_ff).
        feedforward_intermediate_act: The non-linear activation function
          (function or string) applied to the feedforward intermediate layer
          and the pooler layer.
        feedforward_intermediate_dropout_prob: The dropout probability for
          feed-forward intermediate layer.
        moe_num_experts: an integer - number of experts in moe layer
        moe_intermediate_size: an integer - size of intermediate layer in each
          expert
        type_vocab_size: The vocabulary size of the `token_type_ids` passed into
          `BertModel`.
        initializer_range: The stdev of the truncated_normal_initializer for
          initializing all weight matrices.
      """
      self.vocab_size = vocab_size
      self.d_model = d_model
      self.position_signal = position_signal
      self.max_position_embeddings = max_position_embeddings
      self.num_blocks = num_blocks
      self.block_layers = block_layers.split(",")
      self.layer_output_dropout_prob = layer_output_dropout_prob
      self.residual_structure = residual_structure
      self.use_bias = use_bias
      self.attention_probs_dropout_prob = attention_probs_dropout_prob
      self.attention_num_heads = attention_num_heads
      self.attention_head_size = attention_head_size
      self.attention_num_key_heads = attention_num_key_heads
      self.attention_key_head_size = attention_key_head_size
      self.attention_num_value_heads = attention_num_value_heads
      self.attention_value_head_size = attention_value_head_size
      self.feedforward_intermediate_size = feedforward_intermediate_size
      self.feedforward_intermediate_act = feedforward_intermediate_act
      self.feedforward_intermediate_dropout_prob = (
          feedforward_intermediate_dropout_prob
      )
      self.moe_num_experts = moe_num_experts
      self.moe_intermediate_size = moe_intermediate_size
      self.type_vocab_size = type_vocab_size
      self.initializer_range = initializer_range
      if self.position_signal not in [
          "embedding",
          "relative_attention_bias",
      ]:
          raise ValueError("unknown position_signal")
      if self.residual_structure not in ["original", "direct"]:
          raise ValueError("unknown residual_structure")
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  import copy
  import json
  import math
  import re
  import mesh_tensorflow
  import mesh_tensorflow.transformer.moe
  import six
  import tensorflow.compat.v1
label_desc: Use image node_id 1 to create a new BertConfig object from inherited base classes: object with example: obj = BertConfig(vocab_size, d_model, position_signal, max_position_embeddings, num_blocks, block_layers, layer_output_dropout_prob, residual_structure, use_bias, attention_num_heads, attention_head_size, attention_num_key_heads, attention_key_head_size, attention_num_value_heads, attention_value_head_size, attention_probs_dropout_prob, feedforward_intermediate_size, feedforward_intermediate_act, feedforward_intermediate_dropout_prob, moe_num_experts, moe_intermediate_size, type_vocab_size, initializer_range)