class_docstr: Encoder based on a user-supplied vocabulary (file or list).
docstr: Initialize vocabulary with tokens from token_generator.
code: def _init_vocab(self, token_generator, add_reserved_tokens=True):
      """Initialize vocabulary with tokens from token_generator."""
  
      self._id_to_token = {}
      non_reserved_start_index = 0
  
      if add_reserved_tokens:
          self._id_to_token.update(enumerate(RESERVED_TOKENS))
          non_reserved_start_index = len(RESERVED_TOKENS)
  
      self._id_to_token.update(
          enumerate(token_generator, start=non_reserved_start_index)
      )
  
      # _token_to_id is the reverse of _id_to_token
      self._token_to_id = dict(
          (v, k) for k, v in six.iteritems(self._id_to_token)
      )
  
imports: import re
  import six
label_desc: Use image node_id 10 for calling the TokenTextEncoder obj's underlying member method code with example usage: obj._init_vocab(token_generator, add_reserved_tokens) without return types