class_docstr: Tokenizes a tensor of UTF-8 string tokens into subword pieces.
  
  It employs the linear (as opposed to quadratic) WordPiece algorithm (see the
  [paper](http://go/arxiv/2012.15524)).
  
  Differences compared to the classic
  [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer)
  are as follows (as of 11/2021):
  
    * `unknown_token` cannot be None or empty. That means if a word is too long
      or cannot be tokenized, FastWordpieceTokenizer always returns
      `unknown_token`. In constrast, the original
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer)
      would return the original word if `unknown_token` is empty or None.
  
    * `unknown_token` must be included in the vocabulary.
  
    * When `unknown_token` is returned, in tokenize_with_offsets(), the result
      end_offset is set to be the length of the original input word. In
      contrast, when `unknown_token` is returned by the original
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer),
      the end_offset is set to be the length of the `unknown_token` string.
  
    * `split_unknown_characters` is not supported.
  
    * `max_chars_per_token` is not used or needed.
  
    * By default the input is assumed to be general text (i.e., sentences), and
      FastWordpieceTokenizer first splits it on whitespaces and punctuations and
      then applies the Wordpiece tokenization (see the parameter
      `no_pretokenization`). If the input already contains single words only,
      please set `no_pretokenization=True` to be consistent with the classic
      [WordpieceTokenizer](https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer).
class_docstr_tok: ["Tokenizes","a","tensor","of","UTF-8","string","tokens","into","subword","pieces",".","It","employs","the","linear","(","as","opposed","to","quadratic",")","WordPiece","algorithm","(","see","the","[","paper","]","(","http",":","\/\/go\/arxiv\/2012.15524",")",")",".","Differences","compared","to","the","classic","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","are","as","follows","(","as","of","11\/2021",")",":","*","`","unknown_token","`","can","not","be","None","or","empty",".","That","means","if","a","word","is","too","long","or","can","not","be","tokenized",",","FastWordpieceTokenizer","always","returns","`","unknown_token","`",".","In","constrast",",","the","original","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","would","return","the","original","word","if","`","unknown_token","`","is","empty","or","None",".","*","`","unknown_token","`","must","be","included","in","the","vocabulary",".","*","When","`","unknown_token","`","is","returned",",","in","tokenize_with_offsets","(",")",",","the","result","end_offset","is","set","to","be","the","length","of","the","original","input","word",".","In","contrast",",","when","`","unknown_token","`","is","returned","by","the","original","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")",",","the","end_offset","is","set","to","be","the","length","of","the","`","unknown_token","`","string",".","*","`","split_unknown_characters","`","is","not","supported",".","*","`","max_chars_per_token","`","is","not","used","or","needed",".","*","By","default","the","input","is","assumed","to","be","general","text","(","i.e.",",","sentences",")",",","and","FastWordpieceTokenizer","first","splits","it","on","whitespaces","and","punctuations","and","then","applies","the","Wordpiece","tokenization","(","see","the","parameter","`","no_pretokenization","`",")",".","If","the","input","already","contains","single","words","only",",","please","set","`","no_pretokenization=True","`","to","be","consistent","with","the","classic","[","WordpieceTokenizer","]","(","https",":","\/\/www.tensorflow.org\/text\/api_docs\/python\/text\/WordpieceTokenizer",")","."]
docstr: Tokenizes a tensor of UTF-8 string tokens further into subword tokens.
  
  ### Example 1, single word tokenization:
  >>> vocab = ["they", "##'", "##re", "the", "great", "##est", "[UNK]"]
  >>> tokenizer = FastWordpieceTokenizer(vocab, token_out_type=tf.string,
  ...                                    no_pretokenization=True)
  >>> tokens = [["they're", "the", "greatest"]]
  >>> subtokens, starts, ends = tokenizer.tokenize_with_offsets(tokens)
  >>> subtokens
  <tf.RaggedTensor [[[b'they', b"##'", b'##re'], [b'the'],
                     [b'great', b'##est']]]>
  >>> starts
  <tf.RaggedTensor [[[0, 4, 5], [0], [0, 5]]]>
  >>> ends
  <tf.RaggedTensor [[[4, 5, 7], [3], [5, 8]]]>
  
  ### Example 2, general text tokenization (pre-tokenization on
  ### punctuation and whitespace followed by WordPiece tokenization):
  >>> vocab = ["they", "##'", "##re", "the", "great", "##est", "[UNK]",
  ...          "'", "re"]
  >>> tokenizer = FastWordpieceTokenizer(vocab, token_out_type=tf.string)
  >>> tokens = [["they're the greatest", "the greatest"]]
  >>> subtokens, starts, ends = tokenizer.tokenize_with_offsets(tokens)
  >>> subtokens
  <tf.RaggedTensor [[[b'they', b"'", b're', b'the', b'great', b'##est'],
                     [b'the', b'great', b'##est']]]>
  >>> starts
  <tf.RaggedTensor [[[0, 4, 5, 8, 12, 17], [0, 4, 9]]]>
  >>> ends
  <tf.RaggedTensor [[[4, 5, 7, 11, 17, 20], [3, 9, 12]]]>
  
  Args:
    input: An N-dimensional `Tensor` or `RaggedTensor` of UTF-8 strings.
  
  Returns:
    A tuple `(tokens, start_offsets, end_offsets)` where:
  
    tokens: is a `RaggedTensor`, where `tokens[i, j]` is the j-th token
        (i.e., wordpiece) for `input[i]` (i.e., the i-th input word). This
        token is either the actual token string content, or the corresponding
        integer id, i.e., the index of that token string in the vocabulary.
        This choice is controlled by the `token_out_type` parameter passed to
        the initializer method.
    start_offsets[i1...iN, j]: is a `RaggedTensor` of the byte offsets
        for the inclusive start of the `jth` token in `input[i1...iN]`.
    end_offsets[i1...iN, j]: is a `RaggedTensor` of the byte offsets for
        the exclusive end of the `jth` token in `input[i`...iN]` (exclusive,
        i.e., first byte after the end of the token).
docstr_tok: ["Tokenizes","a","tensor","of","UTF-8","string","tokens","further","into","subword","tokens",".","#","#","#","Example","1",",","single","word","tokenization",":",">",">",">","vocab","=","[","``","they","''",",","``","#","#","'","''",",","``","#","#","re","''",",","``","the","''",",","``","great","''",",","``","#","#","est","''",",","``","[","UNK","]","''","]",">",">",">","tokenizer","=","FastWordpieceTokenizer","(","vocab",",","token_out_type=tf.string",",","...","no_pretokenization=True",")",">",">",">","tokens","=","[","[","``","they","'re","''",",","``","the","''",",","``","greatest","''","]","]",">",">",">","subtokens",",","starts",",","ends","=","tokenizer.tokenize_with_offsets","(","tokens",")",">",">",">","subtokens","<","tf.RaggedTensor","[","[","[","b'they","'",",","b","''","#","#","'","''",",","b","'","#","#","re","'","]",",","[","b'the","'","]",",","[","b'great","'",",","b","'","#","#","est","'","]","]","]",">",">",">",">","starts","<","tf.RaggedTensor","[","[","[","0",",","4",",","5","]",",","[","0","]",",","[","0",",","5","]","]","]",">",">",">",">","ends","<","tf.RaggedTensor","[","[","[","4",",","5",",","7","]",",","[","3","]",",","[","5",",","8","]","]","]",">","#","#","#","Example","2",",","general","text","tokenization","(","pre-tokenization","on","#","#","#","punctuation","and","whitespace","followed","by","WordPiece","tokenization",")",":",">",">",">","vocab","=","[","``","they","''",",","``","#","#","'","''",",","``","#","#","re","''",",","``","the","''",",","``","great","''",",","``","#","#","est","''",",","``","[","UNK","]","''",",","...","``","'","''",",","``","re","''","]",">",">",">","tokenizer","=","FastWordpieceTokenizer","(","vocab",",","token_out_type=tf.string",")",">",">",">","tokens","=","[","[","``","they","'re","the","greatest","''",",","``","the","greatest","''","]","]",">",">",">","subtokens",",","starts",",","ends","=","tokenizer.tokenize_with_offsets","(","tokens",")",">",">",">","subtokens","<","tf.RaggedTensor","[","[","[","b'they","'",",","b","''","'","''",",","b","'re","'",",","b'the","'",",","b'great","'",",","b","'","#","#","est","'","]",",","[","b'the","'",",","b'great","'",",","b","'","#","#","est","'","]","]","]",">",">",">",">","starts","<","tf.RaggedTensor","[","[","[","0",",","4",",","5",",","8",",","12",",","17","]",",","[","0",",","4",",","9","]","]","]",">",">",">",">","ends","<","tf.RaggedTensor","[","[","[","4",",","5",",","7",",","11",",","17",",","20","]",",","[","3",",","9",",","12","]","]","]",">","Args",":","input",":","An","N-dimensional","`","Tensor","`","or","`","RaggedTensor","`","of","UTF-8","strings",".","Returns",":","A","tuple","`","(","tokens",",","start_offsets",",","end_offsets",")","`","where",":","tokens",":","is","a","`","RaggedTensor","`",",","where","`","tokens","[","i",",","j","]","`","is","the","j-th","token","(","i.e.",",","wordpiece",")","for","`","input","[","i","]","`","(","i.e.",",","the","i-th","input","word",")",".","This","token","is","either","the","actual","token","string","content",",","or","the","corresponding","integer","id",",","i.e.",",","the","index","of","that","token","string","in","the","vocabulary",".","This","choice","is","controlled","by","the","`","token_out_type","`","parameter","passed","to","the","initializer","method",".","start_offsets","[","i1","...","iN",",","j","]",":","is","a","`","RaggedTensor","`","of","the","byte","offsets","for","the","inclusive","start","of","the","`","jth","`","token","in","`","input","[","i1","...","iN","]","`",".","end_offsets","[","i1","...","iN",",","j","]",":","is","a","`","RaggedTensor","`","of","the","byte","offsets","for","the","exclusive","end","of","the","`","jth","`","token","in","`","input","[","i","`","...","iN","]","`","(","exclusive",",","i.e.",",","first","byte","after","the","end","of","the","token",")","."]
code: def tokenize_with_offsets(
      self, input
  ):  # pylint: disable=redefined-builtin
      """Tokenizes a tensor of UTF-8 string tokens further into subword tokens.
  
      ### Example 1, single word tokenization:
      >>> vocab = ["they", "##'", "##re", "the", "great", "##est", "[UNK]"]
      >>> tokenizer = FastWordpieceTokenizer(vocab, token_out_type=tf.string,
      ...                                    no_pretokenization=True)
      >>> tokens = [["they're", "the", "greatest"]]
      >>> subtokens, starts, ends = tokenizer.tokenize_with_offsets(tokens)
      >>> subtokens
      <tf.RaggedTensor [[[b'they', b"##'", b'##re'], [b'the'],
                         [b'great', b'##est']]]>
      >>> starts
      <tf.RaggedTensor [[[0, 4, 5], [0], [0, 5]]]>
      >>> ends
      <tf.RaggedTensor [[[4, 5, 7], [3], [5, 8]]]>
  
      ### Example 2, general text tokenization (pre-tokenization on
      ### punctuation and whitespace followed by WordPiece tokenization):
      >>> vocab = ["they", "##'", "##re", "the", "great", "##est", "[UNK]",
      ...          "'", "re"]
      >>> tokenizer = FastWordpieceTokenizer(vocab, token_out_type=tf.string)
      >>> tokens = [["they're the greatest", "the greatest"]]
      >>> subtokens, starts, ends = tokenizer.tokenize_with_offsets(tokens)
      >>> subtokens
      <tf.RaggedTensor [[[b'they', b"'", b're', b'the', b'great', b'##est'],
                         [b'the', b'great', b'##est']]]>
      >>> starts
      <tf.RaggedTensor [[[0, 4, 5, 8, 12, 17], [0, 4, 9]]]>
      >>> ends
      <tf.RaggedTensor [[[4, 5, 7, 11, 17, 20], [3, 9, 12]]]>
  
      Args:
        input: An N-dimensional `Tensor` or `RaggedTensor` of UTF-8 strings.
  
      Returns:
        A tuple `(tokens, start_offsets, end_offsets)` where:
  
        tokens: is a `RaggedTensor`, where `tokens[i, j]` is the j-th token
            (i.e., wordpiece) for `input[i]` (i.e., the i-th input word). This
            token is either the actual token string content, or the corresponding
            integer id, i.e., the index of that token string in the vocabulary.
            This choice is controlled by the `token_out_type` parameter passed to
            the initializer method.
        start_offsets[i1...iN, j]: is a `RaggedTensor` of the byte offsets
            for the inclusive start of the `jth` token in `input[i1...iN]`.
        end_offsets[i1...iN, j]: is a `RaggedTensor` of the byte offsets for
            the exclusive end of the `jth` token in `input[i`...iN]` (exclusive,
            i.e., first byte after the end of the token).
      """
      name = None
      with ops.name_scope(
          name, "FastWordpieceTokenizeWithOffsets", [input, self._model]
      ):
          # Check that the types are expected and the ragged rank is appropriate.
          tokens = ragged_tensor.convert_to_tensor_or_ragged_tensor(
              input
          )
          rank = tokens.shape.ndims
          if rank is None:
              raise ValueError("input must have a known rank.")
  
          if rank == 0:
              wordpieces, starts, ends = self.tokenize_with_offsets(
                  array_ops_stack.stack([tokens])
              )
              return wordpieces.values, starts.values, ends.values
  
          elif rank > 1:
              if not ragged_tensor.is_ragged(tokens):
                  tokens = ragged_tensor.RaggedTensor.from_tensor(
                      tokens, ragged_rank=rank - 1
                  )
              wordpieces, starts, ends = self.tokenize_with_offsets(
                  tokens.flat_values
              )
              wordpieces = wordpieces.with_row_splits_dtype(
                  tokens.row_splits.dtype
              )
              starts = starts.with_row_splits_dtype(
                  tokens.row_splits.dtype
              )
              ends = ends.with_row_splits_dtype(tokens.row_splits.dtype)
              return (
                  tokens.with_flat_values(wordpieces),
                  tokens.with_flat_values(starts),
                  tokens.with_flat_values(ends),
              )
  
          # Tokenize the tokens into subwords.
          # TODO(xysong): Optimize below by calling different overload kernels.
          (
              subwords,
              subword_ids,
              row_splits,
              starts,
              ends,
          ) = gen_fast_wordpiece_tokenizer.fast_wordpiece_tokenize_with_offsets(
              input_values=tokens, wp_model=self._model
          )
  
          if self._token_out_type == dtypes.int64:
              values = math_ops.cast(subword_ids, dtypes.int64)
          elif self._token_out_type == dtypes.int32:
              values = math_ops.cast(subword_ids, dtypes.int32)
          else:
              values = subwords
  
          wordpieces = RaggedTensor.from_row_splits(
              values, row_splits, validate=False
          )
          starts = RaggedTensor.from_row_splits(
              starts, row_splits, validate=False
          )
          ends = RaggedTensor.from_row_splits(
              ends, row_splits, validate=False
          )
  
          return wordpieces, starts, ends
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import constant_op
  from tensorflow.python.framework import dtypes
  from tensorflow.python.framework import ops
  from tensorflow.python.framework import tensor
  from tensorflow.python.ops import array_ops_stack
  from tensorflow.python.ops import math_ops
  from tensorflow.python.ops.ragged import ragged_tensor
  from tensorflow.python.ops.ragged.ragged_tensor import RaggedTensor
  from tensorflow_text.core.pybinds import pywrap_fast_wordpiece_tokenizer_model_builder
  from tensorflow_text.python.ops.tokenization import Detokenizer
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
  from tensorflow.python.framework import load_library
  from tensorflow.python.platform import resource_loader
label_desc: Use image node_id 3 for calling the FastWordpieceTokenizer obj's underlying member method code with example usage: obj.tokenize_with_offsets(input) and returns: wordpieces,  starts,  ends, wordpieces,  starts,  ends, tokens,  tokens,  tokens