docstr: Subword tokenization should still run efficiently with long tokens.
  
  To make it run efficiently, we need to use the `max_subtoken_length`
  argument when calling SubwordTextEncoder.build_to_target_size.
docstr_tok: ["Subword","tokenization","should","still","run","efficiently","with","long","tokens",".","To","make","it","run","efficiently",",","we","need","to","use","the","`","max_subtoken_length","`","argument","when","calling","SubwordTextEncoder.build_to_target_size","."]
code: def test_long_tokens(self):
      """Subword tokenization should still run efficiently with long tokens.
  
      To make it run efficiently, we need to use the `max_subtoken_length`
      argument when calling SubwordTextEncoder.build_to_target_size.
      """
      token_length = 4000
      num_tokens = 50
      target_vocab_size = 600
      max_subtoken_length = 10  # Set this to `None` to get problems.
      max_count = 500
  
      # Generate some long random strings.
      random.seed(0)
      long_tokens = []
      for _ in range(num_tokens):
          long_token = "".join(
              [
                  random.choice(string.ascii_uppercase)
                  for _ in range(token_length)
              ]
          )
          long_tokens.append(long_token)
  
      corpus = " ".join(long_tokens)
      token_counts = collections.Counter(corpus.split(" "))
      alphabet = set(corpus) - {" "}
  
      encoder = text_encoder.SubwordTextEncoder.build_to_target_size(
          target_vocab_size,
          token_counts,
          1,
          max_count,
          num_iterations=1,
          max_subtoken_length=max_subtoken_length,
      )
  
      # All vocabulary elements are in the alphabet and subtoken strings even
      # if we requested a smaller vocabulary to assure all expected strings
      # are encodable.
      self.assertTrue(alphabet.issubset(encoder._alphabet))
      for a in alphabet:
          self.assertIn(a, encoder.all_subtoken_strings)
  
imports: import collections
  import io
  import os
  import random
  import shutil
  import string
  import mock
  from six.moves import range
  import tensorflow.compat.v1
  from trax.data import text_encoder
label_desc: Use image node_id 5 for calling the SubwordTextEncoderTest obj's underlying member method code with example usage: obj.test_long_tokens() without return types