class_docstr: Tokenizes a tensor of UTF-8 string tokens into subword pieces.
  
  It employs the linear (as opposed to quadratic) WordPiece algorithm (see the
  [paper](http://go/arxiv/2012.15524)).
  
  Differences compared to the classic
  [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer)
  are as follows (as of 11/2021):
  
    * `unknown_token` cannot be None or empty. That means if a word is too long
      or cannot be tokenized, FastWordpieceTokenizer always returns
      `unknown_token`. In constrast, the original
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer)
      would return the original word if `unknown_token` is empty or None.
  
    * `unknown_token` must be included in the vocabulary.
  
    * When `unknown_token` is returned, in tokenize_with_offsets(), the result
      end_offset is set to be the length of the original input word. In
      contrast, when `unknown_token` is returned by the original
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer),
      the end_offset is set to be the length of the `unknown_token` string.
  
    * `split_unknown_characters` is not supported.
  
    * `max_chars_per_token` is not used or needed.
  
    * By default the input is assumed to be general text (i.e., sentences), and
      FastWordpieceTokenizer first splits it on whitespaces and punctuations and
      then applies the Wordpiece tokenization (see the parameter
      `no_pretokenization`). If the input already contains single words only,
      please set `no_pretokenization=True` to be consistent with the classic
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer).
class_docstr_tok: ["Tokenizes","a","tensor","of","UTF-8","string","tokens","into","subword","pieces",".","It","employs","the","linear","(","as","opposed","to","quadratic",")","WordPiece","algorithm","(","see","the","[","paper","]","(","http",":","\/\/go\/arxiv\/2012.15524",")",")",".","Differences","compared","to","the","classic","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","are","as","follows","(","as","of","11\/2021",")",":","*","`","unknown_token","`","can","not","be","None","or","empty",".","That","means","if","a","word","is","too","long","or","can","not","be","tokenized",",","FastWordpieceTokenizer","always","returns","`","unknown_token","`",".","In","constrast",",","the","original","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","would","return","the","original","word","if","`","unknown_token","`","is","empty","or","None",".","*","`","unknown_token","`","must","be","included","in","the","vocabulary",".","*","When","`","unknown_token","`","is","returned",",","in","tokenize_with_offsets","(",")",",","the","result","end_offset","is","set","to","be","the","length","of","the","original","input","word",".","In","contrast",",","when","`","unknown_token","`","is","returned","by","the","original","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")",",","the","end_offset","is","set","to","be","the","length","of","the","`","unknown_token","`","string",".","*","`","split_unknown_characters","`","is","not","supported",".","*","`","max_chars_per_token","`","is","not","used","or","needed",".","*","By","default","the","input","is","assumed","to","be","general","text","(","i.e.",",","sentences",")",",","and","FastWordpieceTokenizer","first","splits","it","on","whitespaces","and","punctuations","and","then","applies","the","Wordpiece","tokenization","(","see","the","parameter","`","no_pretokenization","`",")",".","If","the","input","already","contains","single","words","only",",","please","set","`","no_pretokenization=True","`","to","be","consistent","with","the","classic","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","."]
docstr: Initializes the FastWordpieceTokenizer.
  
  Two ways to initialize:
    * (preferred) use a precompiled `model_buffer`.
    * use `vocab`, `suffix_indicator`, `max_bytes_per_word`, `unknown_token`,
      and `no_pretokenization`.
  
  Args:
    vocab: (optional) The list of tokens in the vocabulary.
    suffix_indicator: (optional) The characters prepended to a wordpiece to
      indicate that it is a suffix to another subword.
    max_bytes_per_word: (optional) Max size of input token.
    token_out_type: (optional) The type of the token to return. This can be
      `tf.int64` or `tf.int32` IDs, or `tf.string` subwords.
    unknown_token: (optional) The string value to substitute for an unknown
      token. It must be included in `vocab`.
    no_pretokenization: (optional) By default, the input is split on
      whitespaces and punctuations before applying the Wordpiece tokenization.
      When true, the input is assumed to be pretokenized already.
    support_detokenization: (optional) Whether to make the tokenizer support
      doing detokenization. Setting it to true expands the size of the model
      flatbuffer. As a reference, when using 120k multilingual BERT WordPiece
      vocab, the flatbuffer's size increases from ~5MB to ~6MB.
    model_buffer: (optional) Bytes object (or a uint8 tf.Tenosr) that contains
      the wordpiece model in flatbuffer format (see
      fast_wordpiece_tokenizer_model.fbs). If not `None`, all other arguments
      (except `token_output_type`) are ignored.
docstr_tok: ["Initializes","the","FastWordpieceTokenizer",".","Two","ways","to","initialize",":","*","(","preferred",")","use","a","precompiled","`","model_buffer","`",".","*","use","`","vocab","`",",","`","suffix_indicator","`",",","`","max_bytes_per_word","`",",","`","unknown_token","`",",","and","`","no_pretokenization","`",".","Args",":","vocab",":","(","optional",")","The","list","of","tokens","in","the","vocabulary",".","suffix_indicator",":","(","optional",")","The","characters","prepended","to","a","wordpiece","to","indicate","that","it","is","a","suffix","to","another","subword",".","max_bytes_per_word",":","(","optional",")","Max","size","of","input","token",".","token_out_type",":","(","optional",")","The","type","of","the","token","to","return",".","This","can","be","`","tf.int64","`","or","`","tf.int32","`","IDs",",","or","`","tf.string","`","subwords",".","unknown_token",":","(","optional",")","The","string","value","to","substitute","for","an","unknown","token",".","It","must","be","included","in","`","vocab","`",".","no_pretokenization",":","(","optional",")","By","default",",","the","input","is","split","on","whitespaces","and","punctuations","before","applying","the","Wordpiece","tokenization",".","When","true",",","the","input","is","assumed","to","be","pretokenized","already",".","support_detokenization",":","(","optional",")","Whether","to","make","the","tokenizer","support","doing","detokenization",".","Setting","it","to","true","expands","the","size","of","the","model","flatbuffer",".","As","a","reference",",","when","using","120k","multilingual","BERT","WordPiece","vocab",",","the","flatbuffer","'s","size","increases","from","~5MB","to","~6MB",".","model_buffer",":","(","optional",")","Bytes","object","(","or","a","uint8","tf.Tenosr",")","that","contains","the","wordpiece","model","in","flatbuffer","format","(","see","fast_wordpiece_tokenizer_model.fbs",")",".","If","not","`","None","`",",","all","other","arguments","(","except","`","token_output_type","`",")","are","ignored","."]
code: def __init__(
      self,
      vocab=None,
      suffix_indicator="##",
      max_bytes_per_word=100,
      token_out_type=dtypes.int64,
      unknown_token="[UNK]",
      no_pretokenization=False,
      support_detokenization=False,
      model_buffer=None,
  ):
      """Initializes the FastWordpieceTokenizer.
  
      Two ways to initialize:
        * (preferred) use a precompiled `model_buffer`.
        * use `vocab`, `suffix_indicator`, `max_bytes_per_word`, `unknown_token`,
          and `no_pretokenization`.
  
      Args:
        vocab: (optional) The list of tokens in the vocabulary.
        suffix_indicator: (optional) The characters prepended to a wordpiece to
          indicate that it is a suffix to another subword.
        max_bytes_per_word: (optional) Max size of input token.
        token_out_type: (optional) The type of the token to return. This can be
          `tf.int64` or `tf.int32` IDs, or `tf.string` subwords.
        unknown_token: (optional) The string value to substitute for an unknown
          token. It must be included in `vocab`.
        no_pretokenization: (optional) By default, the input is split on
          whitespaces and punctuations before applying the Wordpiece tokenization.
          When true, the input is assumed to be pretokenized already.
        support_detokenization: (optional) Whether to make the tokenizer support
          doing detokenization. Setting it to true expands the size of the model
          flatbuffer. As a reference, when using 120k multilingual BERT WordPiece
          vocab, the flatbuffer's size increases from ~5MB to ~6MB.
        model_buffer: (optional) Bytes object (or a uint8 tf.Tenosr) that contains
          the wordpiece model in flatbuffer format (see
          fast_wordpiece_tokenizer_model.fbs). If not `None`, all other arguments
          (except `token_output_type`) are ignored.
      """
      super(FastWordpieceTokenizer, self).__init__()
      _tf_text_fast_wordpiece_tokenizer_op_create_counter.get_cell().increase_by(
          1
      )
  
      if model_buffer is None:
          model_buffer = pywrap_fast_wordpiece_tokenizer_model_builder.build_fast_wordpiece_model(
              vocab,
              max_bytes_per_word,
              suffix_indicator,
              unknown_token,
              no_pretokenization,
              support_detokenization,
          )
      # Use uint8 tensor as a buffer for the model to avoid any possible changes,
      # for example truncation by '\0'.
      if isinstance(model_buffer, tensor.Tensor):
          self._model = model_buffer
      else:
          self._model = constant_op.constant(
              list(model_buffer), dtype=dtypes.uint8
          )
  
      self._token_out_type = token_out_type
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import constant_op
  from tensorflow.python.framework import dtypes
  from tensorflow.python.framework import ops
  from tensorflow.python.framework import tensor
  from tensorflow.python.ops import array_ops_stack
  from tensorflow.python.ops import math_ops
  from tensorflow.python.ops.ragged import ragged_tensor
  from tensorflow.python.ops.ragged.ragged_tensor import RaggedTensor
  from tensorflow_text.core.pybinds import pywrap_fast_wordpiece_tokenizer_model_builder
  from tensorflow_text.python.ops.tokenization import Detokenizer
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
  from tensorflow.python.framework import load_library
  from tensorflow.python.platform import resource_loader
label_desc: Use image node_id 1 to create a new FastWordpieceTokenizer object from inherited base classes: TokenizerWithOffsets, Detokenizer with example: obj = FastWordpieceTokenizer(vocab, suffix_indicator, max_bytes_per_word, token_out_type, unknown_token, no_pretokenization, support_detokenization, model_buffer)