code: def prep_and_tokenize_generator(
      string_iterable,
      encoder,
      normalize_with_ftfy,
      normalize_with_wikitext_detokenize,
  ):
      """
      Given strings, does data cleaning / tokenization and yields arrays of tokens
      """
      for doc in string_iterable:
          if normalize_with_ftfy:  # fix text with ftfy if specified
              doc = ftfy.fix_text(doc, normalization="NFKC")
          if normalize_with_wikitext_detokenize:
              doc = wikitext_detokenizer(doc)
          tokens = encoder.encode(doc) + [encoder.eos_token_id]
          yield tokens
  
imports: import argparse
  import os
  import re
  import random
  from pathlib import Path
  from typing import List
  import ftfy
  import tensorflow
  from lm_dataformat import Reader
  from transformers import GPT2TokenizerFast
  from tqdm import tqdm
label_desc: Use image node_id 10 for calling a global function with example usage: prep_and_tokenize_generator(string_iterable, encoder, normalize_with_ftfy, normalize_with_wikitext_detokenize) without return types