class_docstr: Invertible `TextEncoder` using word pieces with a byte-level fallback.
  
  Encoding is fully invertible because all out-of-vocab wordpieces are
  byte-encoded.
  
  The vocabulary is "trained" on a corpus and all wordpieces are stored in a
  vocabulary file. To generate a vocabulary from a corpus, use
  `tfds.deprecated.text.SubwordTextEncoder.build_from_corpus`.
  
  Typical usage:
  
  ```
  # Build
  encoder = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(
      corpus_generator, target_vocab_size=2**15)
  encoder.save_to_file(vocab_fname)
  
  # Load
  encoder = tfds.deprecated.text.SubwordTextEncoder.load_from_file(vocab_fname)
  ids = encoder.encode("hello world")
  text = encoder.decode([1, 2, 3, 4])
  ```
docstr: Builds a `SubwordTextEncoder` based on the `corpus_generator`.
  
  Args:
    corpus_generator: generator yielding `str`, from which subwords will be
      constructed.
    target_vocab_size: `int`, approximate size of the vocabulary to create.
    max_subword_length: `int`, maximum length of a subword. Note that memory
      and compute scale quadratically in the length of the longest token.
    max_corpus_chars: `int`, the maximum number of characters to consume from
      `corpus_generator` for the purposes of building the subword vocabulary.
    reserved_tokens: `list<str>`, list of tokens that will always be treated
      as whole tokens and not split up. Note that these must contain a mix of
      alphanumeric and non-alphanumeric characters (e.g. "<EOS>") and not end
      in an underscore.
  
  Returns:
    `SubwordTextEncoder`.
docstr_tok: ["Builds","a","`","SubwordTextEncoder","`","based","on","the","`","corpus_generator","`",".","Args",":","corpus_generator",":","generator","yielding","`","str","`",",","from","which","subwords","will","be","constructed",".","target_vocab_size",":","`","int","`",",","approximate","size","of","the","vocabulary","to","create",".","max_subword_length",":","`","int","`",",","maximum","length","of","a","subword",".","Note","that","memory","and","compute","scale","quadratically","in","the","length","of","the","longest","token",".","max_corpus_chars",":","`","int","`",",","the","maximum","number","of","characters","to","consume","from","`","corpus_generator","`","for","the","purposes","of","building","the","subword","vocabulary",".","reserved_tokens",":","`","list","<","str",">","`",",","list","of","tokens","that","will","always","be","treated","as","whole","tokens","and","not","split","up",".","Note","that","these","must","contain","a","mix","of","alphanumeric","and","non-alphanumeric","characters","(","e.g",".","``","<","EOS",">","''",")","and","not","end","in","an","underscore",".","Returns",":","`","SubwordTextEncoder","`","."]
code: def build_from_corpus(
      cls,
      corpus_generator,
      target_vocab_size,
      max_subword_length=20,
      max_corpus_chars=None,
      reserved_tokens=None,
  ):
      """Builds a `SubwordTextEncoder` based on the `corpus_generator`.
  
      Args:
        corpus_generator: generator yielding `str`, from which subwords will be
          constructed.
        target_vocab_size: `int`, approximate size of the vocabulary to create.
        max_subword_length: `int`, maximum length of a subword. Note that memory
          and compute scale quadratically in the length of the longest token.
        max_corpus_chars: `int`, the maximum number of characters to consume from
          `corpus_generator` for the purposes of building the subword vocabulary.
        reserved_tokens: `list<str>`, list of tokens that will always be treated
          as whole tokens and not split up. Note that these must contain a mix of
          alphanumeric and non-alphanumeric characters (e.g. "<EOS>") and not end
          in an underscore.
  
      Returns:
        `SubwordTextEncoder`.
      """
      reserved_tokens = reserved_tokens or []
      _validate_build_arguments(
          max_subword_length=max_subword_length,
          reserved_tokens=reserved_tokens,
          target_vocab_size=target_vocab_size,
      )
      token_counts = _token_counts_from_generator(
          generator=corpus_generator,
          max_chars=max_corpus_chars,
          reserved_tokens=reserved_tokens,
      )
  
      # Binary search on the minimum token count to build a vocabulary with
      # approximately the right size
      def _binary_search(min_token_count, max_token_count):
          """Binary search min_token_count to build SubwordTextEncoder vocab."""
          candidate_min = (min_token_count + max_token_count) // 2
          logging.info(
              "SubwordTextEncoder build: trying min_token_count %d",
              candidate_min,
          )
          encoder = cls._build_from_token_counts(
              token_counts=token_counts,
              min_token_count=candidate_min,
              reserved_tokens=reserved_tokens,
              num_iterations=4,
              max_subword_length=max_subword_length,
          )
          vocab_size = encoder.vocab_size
  
          # Being within 1% of the target vocab size is ok
          target_achieved = (
              abs(vocab_size - target_vocab_size) * 100
              < target_vocab_size
          )
          if (
              target_achieved
              or min_token_count >= max_token_count
              or candidate_min <= 1
          ):
              # Search complete
              return encoder
  
          # Recurse
          if vocab_size > target_vocab_size:
              next_encoder = _binary_search(
                  candidate_min + 1, max_token_count
              )
          else:
              next_encoder = _binary_search(
                  min_token_count, candidate_min - 1
              )
  
          # Return the one that's closest to the target_vocab_size
          if abs(vocab_size - target_vocab_size) < abs(
              next_encoder.vocab_size - target_vocab_size
          ):
              return encoder
          else:
              return next_encoder
  
      # Get min and max token counts.
      min_token_count = max(min(token_counts.values()), 1)
      max_token_count = max(token_counts.values())
  
      # Another option could be to do a binary search over *ranks* of the tokens.
      return _binary_search(min_token_count, max_token_count)
  
imports: from __future__ import unicode_literals
  import collections
  from absl import logging
  from tensorflow_datasets.core.deprecated.text import text_encoder
  from tensorflow_datasets.core.utils.lazy_imports_utils import tensorflow
label_desc: Use image node_id 14 for calling the SubwordTextEncoder obj's underlying member method code with example usage: obj.build_from_corpus(cls, corpus_generator, target_vocab_size, max_subword_length, max_corpus_chars, reserved_tokens) and returns: _binary_search, encoder, encoder, next_encoder