class_docstr: Expands list of tokens and computes intermediate metrics.
code: def process(self, record):
      wordpiece_counter = collections.Counter(record["wordpieces"])
      del wordpiece_counter["[UNK]"]
      dropped = record["num_dropped_chars"]
      preserved = record["num_preserved_chars"]
      non_unk = record["num_non_unk_wordpieces"]
      preserved_ratio = [preserved / non_unk] if non_unk else []
      dropped_ratio = (
          [dropped / (dropped + preserved)]
          if (dropped + preserved)
          else []
      )
      tokenization_info = {
          "lang": record["lang"],
          "count": 1,
          "num_preserved_chars": preserved,
          "num_dropped_chars": dropped,
          "num_non_unk_wordpieces": non_unk,
          "preserved_ratio": preserved_ratio,
          "dropped_ratio": dropped_ratio,
          "wordpieces": wordpiece_counter,
      }
      yield tokenization_info
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  import apache_beam
  import tensorflow.compat.v1
  from tensorflow_text.python.ops.bert_tokenizer import BertTokenizer
  from tensorflow_text.python.ops.wordpiece_tokenizer import WordpieceTokenizer
  from tensorflow_text.tools.wordpiece_vocab import wordpiece_tokenizer_learner_lib
label_desc: Use image node_id 1 for calling the CompileTokenizationInfo obj's underlying member method code with example usage: obj.process(record) without return types