class_docstr: Automatically trains tokenizers and preprocesses machine translation data based on the MTEncDecModelConfig.
  For training NMT models with datasets larger than 5M sentence pairs, 
  it can be inefficient to train them without first creating a tarred dataset. 
  If the user wants to change the tokenizer, vocab size, or batch size, for example, 
  they must reprocess the data with the correct configuration. 
  With MTDataPreproc users can sweep through data configurations and the tarred dataset will 
  be automatically created according to the model configuration.
  To train tokenizer model and create tarred dataset specify in configuration:
      model.preproc_out_dir=/path/to/preproc_out
      model.encoder_tokenizer.vocab_size=32000
      model.decoder_tokenizer.vocab_size=32000 
      model.train_ds.use_tarred_dataset=True 
      model.train_ds.src_file_name=/path/to/src.txt
      model.train_ds.tgt_file_name=/path/to/tgt.txt
      model.train_ds.tokens_in_batch=16000 
  Once a dataset has been constructed based on this configuration, MTDataPreproc will not process it again.
  If a previously trained tokenizer model or tarred dataset is found, MTDataPreproc will not preprocess the data.
  
  Note: the only tokenizer currently supported is YouTokenToMe.
docstr: Tar files are generated in the following format: basename.number.tar
      Where number is an integer from 1 to the number of tar files.
      We convert this list to a string that can be used in the model config to specify
      tarred datasets: basename_OP_1..num_tar_files_CL_.tar 
  
  Args:
      tar_files (List[str]): List of tar files generated by preprocess_parallel_dataset
docstr_tok: ["Tar","files","are","generated","in","the","following","format",":","basename.number.tar","Where","number","is","an","integer","from","1","to","the","number","of","tar","files",".","We","convert","this","list","to","a","string","that","can","be","used","in","the","model","config","to","specify","tarred","datasets",":","basename_OP_1","..","num_tar_files_CL_.tar","Args",":","tar_files","(","List","[","str","]",")",":","List","of","tar","files","generated","by","preprocess_parallel_dataset"]
code: def tar_files_to_string(self, tar_files):
      """Tar files are generated in the following format: basename.number.tar
          Where number is an integer from 1 to the number of tar files.
          We convert this list to a string that can be used in the model config to specify
          tarred datasets: basename_OP_1..num_tar_files_CL_.tar
  
      Args:
          tar_files (List[str]): List of tar files generated by preprocess_parallel_dataset
      """
      num_tar_files = len(tar_files)
      split_on_dot = tar_files[0].split(".")
      basename = ".".join(split_on_dot[0:-2])
      tar_file_string = f"{basename}._OP_1..{num_tar_files}_CL_.tar"
      return tar_file_string
  
imports: import glob
  import json
  import os
  import pickle
  import tarfile
  import tempfile
  import youtokentome
  from joblib import Parallel, delayed
  from omegaconf import ListConfig, OmegaConf
  from pytorch_lightning import Trainer
  from nemo.collections.common.tokenizers.sentencepiece_tokenizer import SentencePieceTokenizer, create_spt_model
  from nemo.collections.nlp.data.language_modeling.sentence_dataset import SentenceDataset
  from nemo.collections.nlp.data.machine_translation.machine_translation_dataset import TranslationDataset
  from nemo.collections.nlp.models.machine_translation.mt_enc_dec_config import MTEncDecModelConfig
  from nemo.collections.nlp.modules.common.tokenizer_utils import get_nmt_tokenizer, get_tokenizer
  from nemo.utils import logging
label_desc: Use image node_id 2 for calling the MTDataPreproc obj's underlying member method code with example usage: obj.tar_files_to_string(tar_files) and returns: tar_file_string