class_docstr: A mixin class to support loading pre-trained checkpoints for modules
  that implement the GPT2 model.
  
  The GPT2 model was proposed in
  `Language Models are Unsupervised Multitask Learners`_
  by `Radford et al.` from OpenAI. It is a unidirectional Transformer model
  pre-trained using the vanilla language modeling objective on a large corpus.
  
  The available GPT2 models are as follows:
  
    * ``gpt2-small``: Small version of GPT-2, 124M parameters.
    * ``gpt2-medium``: Medium version of GPT-2, 355M parameters.
    * ``gpt2-large``: Large version of GPT-2, 774M parameters.
    * ``gpt2-xl``: XL version of GPT-2, 1558M parameters.
  
  We provide the following GPT2 classes:
  
    * :class:`~texar.tf.modules.GPT2Encoder` for text encoding.
    * :class:`~texar.tf.modules.GPT2Decoder` for text generation and
      decoding.
    * :class:`~texar.tf.modules.GPT2Classifier` for text classification and
      sequence tagging.
  
  .. _`Language Models are Unsupervised Multitask Learners`:
      https://openai.com/blog/better-language-models/
class_docstr_tok: ["A","mixin","class","to","support","loading","pre-trained","checkpoints","for","modules","that","implement","the","GPT2","model",".","The","GPT2","model","was","proposed","in","`","Language","Models","are","Unsupervised","Multitask","Learners","`","_","by","`","Radford","et","al.","`","from","OpenAI",".","It","is","a","unidirectional","Transformer","model","pre-trained","using","the","vanilla","language","modeling","objective","on","a","large","corpus",".","The","available","GPT2","models","are","as","follows",":","*","``","gpt2-small","``",":","Small","version","of","GPT-2",",","124M","parameters",".","*","``","gpt2-medium","``",":","Medium","version","of","GPT-2",",","355M","parameters",".","*","``","gpt2-large","``",":","Large","version","of","GPT-2",",","774M","parameters",".","*","``","gpt2-xl","``",":","XL","version","of","GPT-2",",","1558M","parameters",".","We","provide","the","following","GPT2","classes",":","*",":","class",":","`","~texar.tf.modules.GPT2Encoder","`","for","text","encoding",".","*",":","class",":","`","~texar.tf.modules.GPT2Decoder","`","for","text","generation","and","decoding",".","*",":","class",":","`","~texar.tf.modules.GPT2Classifier","`","for","text","classification","and","sequence","tagging",".","..","_","`","Language","Models","are","Unsupervised","Multitask","Learners","`",":","https",":","\/\/openai.com\/blog\/better-language-models\/"]
code: def _transform_config(
      self, pretrained_model_name: str, cache_dir: str
  ) -> Dict[str, Any]:
      info = list(os.walk(cache_dir))
      root, _, files = info[0]
      config_path = None
      for file in files:
          if file.endswith("hparams.json"):
              config_path = os.path.join(root, file)
      if config_path is None:
          raise ValueError(
              f"Cannot find the config file in {cache_dir}"
          )
  
      with open(config_path) as f:
          config_gpt = json.loads(f.read())
  
      hidden_dim = config_gpt["n_embd"]
      configs = {
          "vocab_size": config_gpt["n_vocab"],
          "context_size": config_gpt["n_ctx"],
          "embedding_size": config_gpt["n_embd"],
          "embed": {
              "dim": hidden_dim,
          },
          "position_size": config_gpt["n_ctx"],
          "position_embed": {"dim": hidden_dim},
      }
  
      module_name = "decoder" if self._IS_DECODE else "encoder"
      configs.update(
          {
              module_name: {
                  "dim": hidden_dim,
                  "num_blocks": config_gpt["n_layer"],
                  "embedding_dropout": 0,
                  "residual_dropout": 0,
                  "multihead_attention": {
                      "use_bias": True,
                      "num_units": hidden_dim,
                      "num_heads": config_gpt["n_head"],
                      "output_dim": hidden_dim,
                  },
                  "initializer": {
                      "type": "variance_scaling_initializer",
                      "kwargs": {
                          "factor": 1.0,
                          "mode": "FAN_AVG",
                          "uniform": True,
                      },
                  },
                  "poswise_feedforward": {
                      "layers": [
                          {
                              "type": "Dense",
                              "kwargs": {
                                  "name": "intermediate",
                                  "activation": "gelu",
                                  "units": hidden_dim * 4,
                                  "use_bias": True,
                              },
                          },
                          {
                              "type": "Dense",
                              "kwargs": {
                                  "activation": None,
                                  "name": "output",
                                  "units": hidden_dim,
                                  "use_bias": True,
                              },
                          },
                      ],
                  },
              }
          }
      )
      return configs
  
file_path: texar/texar/tf/modules/pretrained/gpt2.py
imports: import collections
  import json
  import os
  import re
  import warnings
  from abc import ABC
  from typing import Any, Dict
  import tensorflow
  import numpy
  from texar.tf.modules.pretrained.pretrained_base import PretrainedMixin
label_desc: Use image node_id 1 for calling the PretrainedGPT2Mixin obj's underlying member method code with example usage: obj._transform_config(pretrained_model_name, cache_dir) and returns: configs