class_docstr: Transformer encoder that applies multi-head self attention for encoding
  sequences.
  
  This module basically stacks
  :class:`~texar.tf.modules.encoders.MultiheadAttentionEncoder`,
  :class:`~texar.tf.modules.FeedForwardNetwork` and residual connections.
  
  This module supports two types of architectures, namely, the standard
  Transformer Encoder architecture first proposed in
  `(Vaswani et al.) "Attention is All You Need"`, and
  the variant first used in `(Devlin et al.)` BERT. See
  :meth:`default_hparams` for the nuance between the two types of
  architectures.
  
  Args:
      hparams (dict or HParams, optional): Hyperparameters. Missing
          hyperparameter will be set to default values. See
          :meth:`default_hparams` for the hyperparameter sturcture and
          default values.
  
  .. document private functions
  .. automethod:: _build
docstr: Returns a dictionary of hyperparameters with default values.
  
  .. code-block:: python
  
      {
          "num_blocks": 6,
          "dim": 512,
          'use_bert_config': False,
          "embedding_dropout": 0.1,
          "residual_dropout": 0.1,
          "poswise_feedforward": default_transformer_poswise_net_hparams,
          'multihead_attention': {
              'name': 'multihead_attention',
              'num_units': 512,
              'output_dim': 512,
              'num_heads': 8,
              'dropout_rate': 0.1,
              'output_dim': 512,
              'use_bias': False,
          },
          "initializer": None,
          "name": "transformer_encoder"
      }
  
  Here:
  
  "num_blocks": int
      Number of stacked blocks.
  
  "dim": int
      Hidden dimension of the encoders.
  
  "use_bert_config": bool
      If `False`, apply the standard Transformer Encoder architecture from
      the original paper `(Vaswani et al.) "Attention is All You Need"`.
      If `True`, apply the Transformer Encoder architecture used in BERT
      `(Devlin et al.)`.
  
      The differences lie in:
  
          1. The standard arch restricts the word embedding of PAD token                    to all zero. The BERT arch does not.
  
          2. The attention bias for padding tokens:                    The standard arch uses `-1e8` for nagative attention mask.                    BERT uses `-1e4` instead.
  
          3. The residual connections between internal tensors:                    In BERT, a residual layer connects the tensors *after*                    layer normalization. In the standard arch, the tensors are                    connected *before* layer normalization.
  
  "embedding_dropout": float
      Dropout rate of the input embedding.
  
  "residual_dropout":  float
      Dropout rate of the residual connections.
  
  "poswise_feedforward": dict
      Hyperparameters for a feed-forward network used in residual
      connections.
      Make sure the dimension of the output tensor is equal to `dim`.
  
      See :func:`~texar.tf.modules.default_transformer_poswise_net_hparams`
      for details.
  
  "multihead_attention": dict
      Hyperparameters for the multihead attention strategy.
      Make sure the "output_dim" in this module is equal to "dim".
      See :func:`~texar.tf.modules.MultiheadAttentionEncoder.default_harams`
      for details.
  
  "initializer": dict, optional
      Hyperparameters of the default initializer that initializes
      variables created in this module.
      See :func:`~texar.tf.core.get_initializer` for details.
  
  "name": str
      Name of the module.
docstr_tok: ["Returns","a","dictionary","of","hyperparameters","with","default","values",".","..","code-block",":",":","python","{","``","num_blocks","''",":","6",",","``","dim","''",":","512",",","'use_bert_config","'",":","False",",","``","embedding_dropout","''",":","0.1",",","``","residual_dropout","''",":","0.1",",","``","poswise_feedforward","''",":","default_transformer_poswise_net_hparams",",","'multihead_attention","'",":","{","'name","'",":","'multihead_attention","'",",","'num_units","'",":","512",",","'output_dim","'",":","512",",","'num_heads","'",":","8",",","'dropout_rate","'",":","0.1",",","'output_dim","'",":","512",",","'use_bias","'",":","False",",","}",",","``","initializer","''",":","None",",","``","name","''",":","``","transformer_encoder","''","}","Here",":","''","num_blocks","''",":","int","Number","of","stacked","blocks",".","``","dim","''",":","int","Hidden","dimension","of","the","encoders",".","``","use_bert_config","''",":","bool","If","`","False","`",",","apply","the","standard","Transformer","Encoder","architecture","from","the","original","paper","`","(","Vaswani","et","al",".",")","``","Attention","is","All","You","Need","''","`",".","If","`","True","`",",","apply","the","Transformer","Encoder","architecture","used","in","BERT","`","(","Devlin","et","al.",")","`",".","The","differences","lie","in",":","1",".","The","standard","arch","restricts","the","word","embedding","of","PAD","token","to","all","zero",".","The","BERT","arch","does","not",".","2",".","The","attention","bias","for","padding","tokens",":","The","standard","arch","uses","`","-1e8","`","for","nagative","attention","mask",".","BERT","uses","`","-1e4","`","instead",".","3",".","The","residual","connections","between","internal","tensors",":","In","BERT",",","a","residual","layer","connects","the","tensors","*","after","*","layer","normalization",".","In","the","standard","arch",",","the","tensors","are","connected","*","before","*","layer","normalization",".","``","embedding_dropout","''",":","float","Dropout","rate","of","the","input","embedding",".","``","residual_dropout","''",":","float","Dropout","rate","of","the","residual","connections",".","``","poswise_feedforward","''",":","dict","Hyperparameters","for","a","feed-forward","network","used","in","residual","connections",".","Make","sure","the","dimension","of","the","output","tensor","is","equal","to","`","dim","`",".","See",":","func",":","`","~texar.tf.modules.default_transformer_poswise_net_hparams","`","for","details",".","``","multihead_attention","''",":","dict","Hyperparameters","for","the","multihead","attention","strategy",".","Make","sure","the","``","output_dim","''","in","this","module","is","equal","to","``","dim","''",".","See",":","func",":","`","~texar.tf.modules.MultiheadAttentionEncoder.default_harams","`","for","details",".","``","initializer","''",":","dict",",","optional","Hyperparameters","of","the","default","initializer","that","initializes","variables","created","in","this","module",".","See",":","func",":","`","~texar.tf.core.get_initializer","`","for","details",".","``","name","''",":","str","Name","of","the","module","."]
code: def default_hparams():
      """Returns a dictionary of hyperparameters with default values.
  
          .. code-block:: python
  
              {
                  "num_blocks": 6,
                  "dim": 512,
                  'use_bert_config': False,
                  "embedding_dropout": 0.1,
                  "residual_dropout": 0.1,
                  "poswise_feedforward": default_transformer_poswise_net_hparams,
                  'multihead_attention': {
                      'name': 'multihead_attention',
                      'num_units': 512,
                      'output_dim': 512,
                      'num_heads': 8,
                      'dropout_rate': 0.1,
                      'output_dim': 512,
                      'use_bias': False,
                  },
                  "initializer": None,
                  "name": "transformer_encoder"
              }
  
          Here:
  
          "num_blocks": int
              Number of stacked blocks.
  
          "dim": int
              Hidden dimension of the encoders.
  
          "use_bert_config": bool
              If `False`, apply the standard Transformer Encoder architecture from
              the original paper `(Vaswani et al.) "Attention is All You Need"`.
              If `True`, apply the Transformer Encoder architecture used in BERT
              `(Devlin et al.)`.
  
              The differences lie in:
  
                  1. The standard arch restricts the word embedding of PAD token \
                     to all zero. The BERT arch does not.
  
                  2. The attention bias for padding tokens: \
                     The standard arch uses `-1e8` for nagative attention mask. \
                     BERT uses `-1e4` instead.
  
                  3. The residual connections between internal tensors: \
                     In BERT, a residual layer connects the tensors *after* \
                     layer normalization. In the standard arch, the tensors are \
                     connected *before* layer normalization.
  
          "embedding_dropout": float
              Dropout rate of the input embedding.
  
          "residual_dropout":  float
              Dropout rate of the residual connections.
  
          "poswise_feedforward": dict
              Hyperparameters for a feed-forward network used in residual
              connections.
              Make sure the dimension of the output tensor is equal to `dim`.
  
              See :func:`~texar.tf.modules.default_transformer_poswise_net_hparams`
              for details.
  
          "multihead_attention": dict
              Hyperparameters for the multihead attention strategy.
              Make sure the "output_dim" in this module is equal to "dim".
              See :func:`~texar.tf.modules.MultiheadAttentionEncoder.default_harams`
              for details.
  
          "initializer": dict, optional
              Hyperparameters of the default initializer that initializes
              variables created in this module.
              See :func:`~texar.tf.core.get_initializer` for details.
  
          "name": str
              Name of the module.
          """
      return {
          "num_blocks": 6,
          "dim": 512,
          "use_bert_config": False,
          "embedding_dropout": 0.1,
          "residual_dropout": 0.1,
          "poswise_feedforward": default_transformer_poswise_net_hparams(),
          "multihead_attention": {
              "name": "multihead_attention",
              "num_units": 512,
              "num_heads": 8,
              "dropout_rate": 0.1,
              "output_dim": 512,
              "use_bias": False,
          },
          "initializer": None,
          "name": "transformer_encoder",
      }
  
file_path: texar/texar/tf/modules/encoders/transformer_encoders.py
imports: import tensorflow
  from texar.tf.core import layers
  from texar.tf.utils import transformer_attentions, transformer_utils
  from texar.tf.modules.encoders.encoder_base import EncoderBase
  from texar.tf.modules.encoders.multihead_attention import MultiheadAttentionEncoder
  from texar.tf.modules.networks.networks import FeedForwardNetwork
  from texar.tf.utils.shapes import shape_list
  from texar.tf.utils.mode import is_train_mode
label_desc: Use image node_id 2 for calling the TransformerEncoder obj's underlying member method code with example usage: obj.default_hparams() and returns: dict