class_docstr: Splits an already-tokenized tensor of tokens further into WordPiece tokens.
  
  Splits a set of string tokens into subwords as described in
  https://arxiv.org/pdf/1609.08144.pdf. This layer does not build the WordPiece
  vocabulary; instead, users should set the vocabulary by either passing it to
  the init call or by calling set_vocabulary() after the layer is constructed.
  
  Attributes:
    vocabulary: An optional list of vocabulary terms, or a path to a text file
      containing a vocabulary to load into this layer. The file should contain
      one token per line. If the list or file contains the same token multiple
      times, an error will be thrown.
    suffix_indicator: (optional) The characters prepended to a wordpiece to
      indicate that it is a suffix to another subword. Default is '##'.
    max_bytes_per_word: (optional) Max size of input token. Default is 100.
    token_out_type: (optional) The type of the token to return. This can be
      `tf.int64` IDs, or `tf.string` subwords. The default is `tf.int64`.
    unknown_token: (optional) The string value to substitute for an unknown
      token. Default is "[UNK]". If set to `None`, no substitution occurs.
      If `token_out_type` is `tf.int64`, the `vocabulary` is used (after
      substitution) to convert the unknown token to an integer, resulting in -1
      if `unknown_token` is set to `None` or not contained in the `vocabulary`.
    pad_value: if not None, performs the padding (using pad_value) at the
      inner-most dimension (i.e. token dimension) and outputs a padded dense
      tensor (default=None).
    merge_wordpiece_dim: If False, this layer will output a RaggedTensor
      with an additional inner 'wordpiece' dimension, containing the wordpieces
      for each token. If set to True, this layer will concatenate and squeeze
      along that dimension. Defaults to True.
code: def __init__(
      self,
      vocabulary=None,
      suffix_indicator="##",
      max_bytes_per_word=100,
      token_out_type=tf.string,
      unknown_token="[UNK]",
      pad_value=None,
      merge_wordpiece_dim=True,
      **kwargs,
  ):
      self._suffix_indicator = suffix_indicator
      self._max_bytes_per_word = max_bytes_per_word
      self._token_out_type = tf.dtypes.as_dtype(token_out_type)
      self._unknown_token = unknown_token
      self._merge_wordpiece_dim = merge_wordpiece_dim
  
      self._table = lookup_ops.MutableHashTable(
          key_dtype=tf.string, value_dtype=tf.int64, default_value=-1
      )
  
      tokenizer_instance = wordpiece_tokenizer.WordpieceTokenizer(
          vocab_lookup_table=self._table,
          suffix_indicator=self._suffix_indicator,
          max_bytes_per_word=self._max_bytes_per_word,
          token_out_type=self._token_out_type,
          unknown_token=self._unknown_token,
      )
  
      super(WordpieceTokenizer, self).__init__(
          tokenizer_instance=tokenizer_instance,
          squeeze_token_dim=False,
          pad_value=pad_value,
          **kwargs,
      )
  
      # We need to add the trackable after the superclass was called, since
      # it adds the table to a list that is created there.
      tracked_table = self._add_trackable(self._table, trainable=False)
      # This is a workaround for summary() on this layer. Because the table is
      # not mutable during training, the effective number of parameters (and so
      # the weight shape) is 0; we add this as an attr so that the parameter
      # counting code in the Model object doesn't throw an attribute error.
      tracked_table.shape = tf.TensorShape((0,))
  
      if vocabulary is not None:
          self.set_vocabulary(vocabulary)
  
imports: import os
  import tensorflow
  from tensorflow.python.ops import lookup_ops
  from tensorflow.python.ops.ragged import ragged_conversion_ops
  from tensorflow_text.python.ops import unicode_script_tokenizer
  from tensorflow_text.python.ops import whitespace_tokenizer
  from tensorflow_text.python.ops import wordpiece_tokenizer
label_desc: Use image node_id 1 to create a new WordpieceTokenizer object from inherited base classes: TokenizerBase with example: obj = WordpieceTokenizer(vocabulary, suffix_indicator, max_bytes_per_word, token_out_type, unknown_token, pad_value, merge_wordpiece_dim)