class_docstr: TextEncoder backed by a list of tokens.
  
  Tokenization splits on (and drops) non-alphanumeric characters with
  regex "\W+".
class_docstr_tok: ["TextEncoder","backed","by","a","list","of","tokens",".","Tokenization","splits","on","(","and","drops",")","non-alphanumeric","characters","with","regex","``","\\W+","''","."]
docstr: Constructs a TokenTextEncoder.
  
  To load from a file saved with `TokenTextEncoder.save_to_file`, use
  `TokenTextEncoder.load_from_file`.
  
  Args:
    vocab_list: `list<str>`, list of tokens.
    oov_buckets: `int`, the number of `int`s to reserve for OOV hash buckets.
      Tokens that are OOV will be hash-modded into a OOV bucket in `encode`.
    oov_token: `str`, the string to use for OOV ids in `decode`.
    lowercase: `bool`, whether to make all text and tokens lowercase.
    tokenizer: `Tokenizer`, responsible for converting incoming text into a
      list of tokens.
    strip_vocab: `bool`, whether to strip whitespace from the beginning and
      end of elements of `vocab_list`.
    decode_token_separator: `str`, the string used to separate tokens when
      decoding.
code: def __init__(
      self,
      vocab_list,
      oov_buckets=1,
      oov_token="UNK",
      lowercase=False,
      tokenizer=None,
      strip_vocab=True,
      decode_token_separator=" ",
  ):
      """Constructs a TokenTextEncoder.
  
      To load from a file saved with `TokenTextEncoder.save_to_file`, use
      `TokenTextEncoder.load_from_file`.
  
      Args:
        vocab_list: `list<str>`, list of tokens.
        oov_buckets: `int`, the number of `int`s to reserve for OOV hash buckets.
          Tokens that are OOV will be hash-modded into a OOV bucket in `encode`.
        oov_token: `str`, the string to use for OOV ids in `decode`.
        lowercase: `bool`, whether to make all text and tokens lowercase.
        tokenizer: `Tokenizer`, responsible for converting incoming text into a
          list of tokens.
        strip_vocab: `bool`, whether to strip whitespace from the beginning and
          end of elements of `vocab_list`.
        decode_token_separator: `str`, the string used to separate tokens when
          decoding.
      """
      self._vocab_list = [tf.compat.as_text(el) for el in vocab_list]
      if strip_vocab:
          self._vocab_list = [el.strip() for el in self._vocab_list]
      self._lowercase = lowercase
      if self._lowercase:
          self._vocab_list = [t.lower() for t in self._vocab_list]
      # Note that internally everything is 0-indexed. Padding is dealt with at the
      # end of encode and the beginning of decode.
      self._token_to_id = dict(
          zip(self._vocab_list, range(len(self._vocab_list)))
      )
      self._oov_buckets = oov_buckets
      self._oov_token = tf.compat.as_text(oov_token)
  
      # Reserved tokens are all tokens that are mixed alphanum and non-alphanum.
      reserved_tokens = [
          t for t in self._vocab_list if is_mixed_alphanum(t)
      ]
      self._tokenizer = tokenizer or Tokenizer(
          reserved_tokens=reserved_tokens
      )
      self._user_defined_tokenizer = tokenizer
  
      self._decode_token_separator = decode_token_separator
  
imports: from __future__ import unicode_literals
  import abc
  import hashlib
  import json
  import re
  from tensorflow_datasets.core.utils.lazy_imports_utils import tensorflow
label_desc: Use image node_id 1 to create a new TokenTextEncoder object from inherited base classes: TextEncoder with example: obj = TokenTextEncoder(vocab_list, oov_buckets, oov_token, lowercase, tokenizer, strip_vocab, decode_token_separator)