class_docstr: Raw GPT2 Transformer for decoding sequences. Please see
  :class:`~texar.tf.modules.PretrainedGPT2Mixin` for a brief description
  of GPT2.
  
  This module basically stacks
  :class:`~texar.tf.modules.WordEmbedder`,
  :class:`~texar.tf.modules.PositionEmbedder`,
  :class:`~texar.tf.modules.TransformerDecoder`.
  
  This module supports the architecture first proposed
  in `(Radford et al.)` GPT2.
  
  Args:
      pretrained_model_name (optional): a `str`, the name
          of pre-trained model (e.g., ``gpt2-small``). Please refer to
          :class:`~texar.tf.modules.PretrainedGPT2Mixin` for
          all supported models.
          If `None`, the model name in :attr:`hparams` is used.
      cache_dir (optional): the path to a folder in which the
          pre-trained models will be cached. If `None` (default),
          a default directory (``texar_data`` folder under user's home
          directory) will be used.
      hparams (dict or HParams, optional): Hyperparameters. Missing
          hyperparameter will be set to default values. See
          :meth:`default_hparams` for the hyperparameter structure
          and default values.
  
  .. document private functions
  .. automethod:: _build
docstr: Returns a dictionary of hyperparameters with default values.
  
  * The decoder arch is determined by the constructor argument
    :attr:`pretrained_model_name` if it's specified. In this case,
    `hparams` are ignored.
  * Otherwise, the encoder arch is determined by
    `hparams['pretrained_model_name']` if it's specified. All other
    configurations in `hparams` are ignored.
  * If the above two are `None`, the decoder arch is defined by the
    configurations in `hparams` and weights are randomly initialized.
  
  .. code-block:: python
  
      {
          "name": "gpt2_decoder",
          "pretrained_model_name": "gpt2-small",
          "vocab_size": 50257,
          "context_size": 1024,
          "embedding_size": 768,
          "embed": {
              "dim": 768,
              "name": "word_embeddings"
          },
          "position_size": 1024,
          "position_embed": {
              "dim": 768,
              "name": "position_embeddings"
          },
  
          # hparams for TransformerDecoder
          "decoder": {
              "dim": 768,
              "num_blocks": 12,
              "use_gpt_config": True,
              "embedding_dropout": 0,
              "residual_dropout": 0,
              "multihead_attention": {
                  "use_bias": True,
                  "num_units": 768,
                  "num_heads": 12,
                  "dropout_rate": 0.0,
                  "output_dim": 768
              },
              "initializer": {
                  "type": "variance_scaling_initializer",
                  "kwargs": {
                      "factor": 1.0,
                      "mode": "FAN_AVG",
                      "uniform": True
                  }
              },
              "poswise_feedforward": {
                  "layers": [
                      {
                          "type": "Dense",
                          "kwargs": {
                              "activation": "gelu",
                              "name": "intermediate",
                              "units": 3072,
                              "use_bias": True
                          }
                      },
                      {
                          "type": "Dense",
                          "kwargs": {
                              "activation": None,
                              "name": "output",
                              "units": 3072,
                              "use_bias": True
                          }
                      }
                  ],
                  "name": "ffn"
              }
          },
          "name": "gpt2_decoder",
      }
  
  Here:
  
  The default parameters are values for 124M GPT2 model.
  
  `"pretrained_model_name"`: str or None
      The name of the pre-trained GPT2 model. If None, the model
      will be randomly initialized.
  
  `"embed"`: dict
      Hyperparameters for word embedding layer.
  
  `"vocab_size"`: int
      The vocabulary size of `inputs` in `GPT2Model`.
  
  `"position_embed"`: dict
      Hyperparameters for position embedding layer.
  
  `"position_size"`:  int
      The maximum sequence length that this model might ever be used with.
  
  `"name"`: str
      Name of the module.
code: def default_hparams():
      r"""Returns a dictionary of hyperparameters with default values.
  
      * The decoder arch is determined by the constructor argument
        :attr:`pretrained_model_name` if it's specified. In this case,
        `hparams` are ignored.
      * Otherwise, the encoder arch is determined by
        `hparams['pretrained_model_name']` if it's specified. All other
        configurations in `hparams` are ignored.
      * If the above two are `None`, the decoder arch is defined by the
        configurations in `hparams` and weights are randomly initialized.
  
      .. code-block:: python
  
          {
              "name": "gpt2_decoder",
              "pretrained_model_name": "gpt2-small",
              "vocab_size": 50257,
              "context_size": 1024,
              "embedding_size": 768,
              "embed": {
                  "dim": 768,
                  "name": "word_embeddings"
              },
              "position_size": 1024,
              "position_embed": {
                  "dim": 768,
                  "name": "position_embeddings"
              },
  
              # hparams for TransformerDecoder
              "decoder": {
                  "dim": 768,
                  "num_blocks": 12,
                  "use_gpt_config": True,
                  "embedding_dropout": 0,
                  "residual_dropout": 0,
                  "multihead_attention": {
                      "use_bias": True,
                      "num_units": 768,
                      "num_heads": 12,
                      "dropout_rate": 0.0,
                      "output_dim": 768
                  },
                  "initializer": {
                      "type": "variance_scaling_initializer",
                      "kwargs": {
                          "factor": 1.0,
                          "mode": "FAN_AVG",
                          "uniform": True
                      }
                  },
                  "poswise_feedforward": {
                      "layers": [
                          {
                              "type": "Dense",
                              "kwargs": {
                                  "activation": "gelu",
                                  "name": "intermediate",
                                  "units": 3072,
                                  "use_bias": True
                              }
                          },
                          {
                              "type": "Dense",
                              "kwargs": {
                                  "activation": None,
                                  "name": "output",
                                  "units": 3072,
                                  "use_bias": True
                              }
                          }
                      ],
                      "name": "ffn"
                  }
              },
              "name": "gpt2_decoder",
          }
  
      Here:
  
      The default parameters are values for 124M GPT2 model.
  
      `"pretrained_model_name"`: str or None
          The name of the pre-trained GPT2 model. If None, the model
          will be randomly initialized.
  
      `"embed"`: dict
          Hyperparameters for word embedding layer.
  
      `"vocab_size"`: int
          The vocabulary size of `inputs` in `GPT2Model`.
  
      `"position_embed"`: dict
          Hyperparameters for position embedding layer.
  
      `"position_size"`:  int
          The maximum sequence length that this model might ever be used with.
  
      `"name"`: str
          Name of the module.
      """
      return {
          "decoder": {
              "name": "decoder",
              "dim": 768,
              "num_blocks": 12,
              "embedding_dropout": 0,
              "residual_dropout": 0,
              "multihead_attention": {
                  "name": "self",
                  "use_bias": True,
                  "num_units": 768,
                  "num_heads": 12,
                  "dropout_rate": 0.0,
                  "output_dim": 768,
              },
              "initializer": {
                  "type": "variance_scaling_initializer",
                  "kwargs": {
                      "factor": 1.0,
                      "mode": "FAN_AVG",
                      "uniform": True,
                  },
              },
              "poswise_feedforward": {
                  "layers": [
                      {
                          "type": "Dense",
                          "kwargs": {
                              "activation": "gelu",
                              "name": "intermediate",
                              "units": 3072,
                              "use_bias": True,
                          },
                      },
                      {
                          "type": "Dense",
                          "kwargs": {
                              "activation": None,
                              "name": "output",
                              "units": 768,
                              "use_bias": True,
                          },
                      },
                  ],
                  "name": "ffn",
              },
          },
          "pretrained_model_name": "gpt2-small",
          "vocab_size": 50257,
          "context_size": 1024,
          "embedding_size": 768,
          "embed": {"dim": 768, "name": "word_embeddings"},
          "position_size": 1024,
          "position_embed": {"dim": 768, "name": "position_embeddings"},
          "name": "gpt2_decoder",
          "@no_typecheck": ["pretrained_model_name"],
      }
  
file_path: texar/texar/tf/modules/decoders/gpt2_decoder.py
imports: import tensorflow
  from texar.tf.modules.decoders.transformer_decoders import TransformerDecoder
  from texar.tf.modules.embedders import PositionEmbedder, WordEmbedder
  from texar.tf.modules.pretrained.gpt2 import PretrainedGPT2Mixin
label_desc: Use image node_id 3 for calling the GPT2Decoder obj's underlying member method code with example usage: obj.default_hparams() and returns: dict